From 851b528ca36cb996b0e9de0b815e6d05364dfc32 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sat, 15 Aug 2026 21:34:17 -0400 Subject: [PATCH] =?UTF-8?q?docs(deployment):=20row=2079=20=E2=80=94=20OrbS?= =?UTF-8?q?tack/virtiofs=20rig=20hazard=20+=20proven=20recovery=20runbook?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents the host-level hazard behind row 69's LocalDb corruption, all live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a property of the filesystem the rig's data files sit on, not a ScadaBridge defect and not something a production Linux deployment is exposed to. Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh uses, is exactly the abrupt mid-write kill that turns that into a torn file. Records both observed failure modes, and specifically how differently they present, which is the part that makes this hard to spot: site-a-b failed loudly while running (every sync call throwing, connected:false, backlog 892), while site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a new build's boot-time re-registration read pages the old build never touched (SIGABRT crash-loop x12). A clean-looking node proves nothing about its file. Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in ~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup` errors and a hung docker until `orb restart docker` (data intact, containers auto-restart). Written so a future session checks vmgr.log BEFORE investigating ScadaBridge, which is where ~20 minutes went last time. Recovery runbook is the one that actually worked: stop the pair together, quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt- suffix (never delete — forensics; and a stale WAL beside a fresh database is its own corruption), restart together, then VERIFY convergence rather than assuming it. Notes the precondition that this is a replicated pair's recovery — it leans on the startup reconcile plus CDC snapshot resync, so an unreplicated node has no peer to reseed from and needs a backup instead. Mitigation options tabulated with their real costs (named volumes = native ext4 but loses the host-side inspectability the debugging sessions lean on; disabling OrbStack idle-pause = a host setting that cannot be version-controlled and silently does not exist on a new machine; accept + runbook = current), so the status quo reads as a choice rather than an oversight. Docs only — no compose change, no host config change, rig untouched. --- docs/deployment/topology-guide.md | 97 +++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/docs/deployment/topology-guide.md b/docs/deployment/topology-guide.md index 5320168c..8836cd8c 100644 --- a/docs/deployment/topology-guide.md +++ b/docs/deployment/topology-guide.md @@ -311,6 +311,103 @@ a rejoin is safe and self-correcting (verified live: a node stopped and restarte with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's database from its peer rather than letting it rejoin. +### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard + +**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a +host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and +not something a production Linux deployment is exposed to. It is documented here because it has +already cost two databases and half a session, and because the recovery is non-obvious. + +Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree +— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`, +`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the +Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety +contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true, +an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate` +— which `deploy.sh` uses — is exactly such an abrupt kill. + +Two distinct failure modes have been observed live (both 2026-08-15): + +**1. Silent database corruption.** Both site-a LocalDb files were found malformed: + +``` +SQLite Error 11: 'database disk image is malformed' +``` + +Note how differently the two nodes presented, because this is the part that makes it hard to spot: + +- **site-a-b** surfaced it as a *running* fault — every sync call threw, replication reported + `connected: false` with a backlog of 892, and the faulted session retried about once a minute. + The node stayed up and otherwise healthy-looking. +- **site-a-a** was corrupt too, but **latently** — nothing on the running build ever read the + damaged pages. It only became fatal on the *next upgrade*, when a new build's boot-time + re-registration read pages the old build never touched: `SqliteLocalDb` construction threw and + the container **SIGABRT crash-looped** (×12). + +The lesson to carry: a clean-looking node proves nothing about its file. Corruption can sit latent +until an unrelated code change happens to read further into the database. + +**2. The OrbStack VM idle-sleeping mid-operation.** The VM can suspend under its idle policy while +work is in flight. `~/.orbstack/log/vmgr.log` records it plainly: + +``` +vmgr ... msg=sleep +``` + +On wake, the virtiofs layer logs repeated errors of the form: + +``` +refresh_nodeid: race with new lookup +``` + +and `docker` hangs — commands stop returning, `docker ps` comes back empty — for as long as it +takes to notice. **The fix is `orb restart docker`**; data survives and containers auto-restart. +Observed cost: roughly 20 minutes of a session lost to diagnosing what looked like a cluster +failure but was a host one. If the rig goes unresponsive as a whole, check `vmgr.log` **before** +investigating ScadaBridge. + +#### Recovery runbook — LocalDb corruption on a site pair (proven) + +This is the procedure that recovered site-a. It relies on the pair being able to reseed itself, +so it is a *replicated* pair's recovery; an unreplicated node has no peer to reseed from and must +be restored from a backup instead. + +1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same + stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason. +2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping + them for forensics: + + ```sh + # per affected node's data directory + mv site-localdb.db site-localdb.db.corrupt-$(date +%F) + mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F) + mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F) + ``` + + The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh + database is its own corruption. +3. **Start both nodes together.** The fresh files reseed themselves from two independent paths: + `SiteReconciliationActor`'s node-startup fetch pulls config back from central when central + reports gaps, and LocalDb's **CDC snapshot resync** merges the peer's rows under LWW. +4. **Verify convergence**, do not assume it: `LocalDbReplicationConnected` should read `true` and + `LocalDbOplogBacklog` should settle to ~0 on the site health report. (See *Reading the + replication backlog* above for why a transient spike is normal and not a fault.) + +On the live recovery, both quarantined files were large (394 MB and 416 MB), the pair converged to +`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the +digest handshake green. + +#### Mitigation options, with their trade-offs + +None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted +position is the runbook above. Recorded so the choice is a choice: + +| Option | Gains | Costs | +|--------|-------|-------| +| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. | +| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. | +| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. | + ### Central-Site Communication Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed