Documents the host-level hazard behind row 69's LocalDb corruption, all
live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a
property of the filesystem the rig's data files sit on, not a ScadaBridge defect
and not something a production Linux deployment is exposed to.
Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative
bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs
does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh
uses, is exactly the abrupt mid-write kill that turns that into a torn file.
Records both observed failure modes, and specifically how differently they
present, which is the part that makes this hard to spot: site-a-b failed loudly
while running (every sync call throwing, connected:false, backlog 892), while
site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a
new build's boot-time re-registration read pages the old build never touched
(SIGABRT crash-loop x12). A clean-looking node proves nothing about its file.
Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in
~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup`
errors and a hung docker until `orb restart docker` (data intact, containers
auto-restart). Written so a future session checks vmgr.log BEFORE investigating
ScadaBridge, which is where ~20 minutes went last time.
Recovery runbook is the one that actually worked: stop the pair together,
quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt-<date>
suffix (never delete — forensics; and a stale WAL beside a fresh database is its
own corruption), restart together, then VERIFY convergence rather than assuming
it. Notes the precondition that this is a replicated pair's recovery — it leans
on the startup reconcile plus CDC snapshot resync, so an unreplicated node has
no peer to reseed from and needs a backup instead.
Mitigation options tabulated with their real costs (named volumes = native ext4
but loses the host-side inspectability the debugging sessions lean on;
disabling OrbStack idle-pause = a host setting that cannot be version-controlled
and silently does not exist on a new machine; accept + runbook = current), so
the status quo reads as a choice rather than an oversight.
Docs only — no compose change, no host config change, rig untouched.
Joseph Doherty
851b528ca3