docs(deployment): row 79 — OrbStack/virtiofs rig hazard + proven recovery runbook

Documents the host-level hazard behind row 69's LocalDb corruption, all
live-verified 2026-08-15. Scope is the macOS docker rigs only — this is a
property of the filesystem the rig's data files sit on, not a ScadaBridge defect
and not something a production Linux deployment is exposed to.

Every rig data file (site-localdb.db, auditlog.db, secrets) rides a relative
bind mount out of the repo tree, which on macOS crosses virtiofs — and virtiofs
does not give SQLite true fsync durability. `--force-recreate`, which deploy.sh
uses, is exactly the abrupt mid-write kill that turns that into a torn file.

Records both observed failure modes, and specifically how differently they
present, which is the part that makes this hard to spot: site-a-b failed loudly
while running (every sync call throwing, connected:false, backlog 892), while
site-a-a was corrupt LATENTLY and only became fatal on the next upgrade, when a
new build's boot-time re-registration read pages the old build never touched
(SIGABRT crash-loop x12). A clean-looking node proves nothing about its file.
Second mode is the VM idle-sleeping mid-operation — `vmgr ... msg=sleep` in
~/.orbstack/log/vmgr.log, then virtiofs `refresh_nodeid: race with new lookup`
errors and a hung docker until `orb restart docker` (data intact, containers
auto-restart). Written so a future session checks vmgr.log BEFORE investigating
ScadaBridge, which is where ~20 minutes went last time.

Recovery runbook is the one that actually worked: stop the pair together,
quarantine site-localdb.db AND its -wal/-shm sidecars with a .corrupt-<date>
suffix (never delete — forensics; and a stale WAL beside a fresh database is its
own corruption), restart together, then VERIFY convergence rather than assuming
it. Notes the precondition that this is a replicated pair's recovery — it leans
on the startup reconcile plus CDC snapshot resync, so an unreplicated node has
no peer to reseed from and needs a backup instead.

Mitigation options tabulated with their real costs (named volumes = native ext4
but loses the host-side inspectability the debugging sessions lean on;
disabling OrbStack idle-pause = a host setting that cannot be version-controlled
and silently does not exist on a new machine; accept + runbook = current), so
the status quo reads as a choice rather than an oversight.

Docs only — no compose change, no host config change, rig untouched.

docs/deployment/topology-guide.md

@@ -311,6 +311,103 @@ a rejoin is safe and self-correcting (verified live: a node stopped and restarte
 with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's
 database from its peer rather than letting it rejoin.
 
+### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard
+
+**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a
+host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and
+not something a production Linux deployment is exposed to. It is documented here because it has
+already cost two databases and half a session, and because the recovery is non-obvious.
+
+Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree
+— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`,
+`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the
+Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety
+contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true,
+an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate`
+— which `deploy.sh` uses — is exactly such an abrupt kill.
+
+Two distinct failure modes have been observed live (both 2026-08-15):
+
+**1. Silent database corruption.** Both site-a LocalDb files were found malformed:
+
+```
+SQLite Error 11: 'database disk image is malformed'
+```
+
+Note how differently the two nodes presented, because this is the part that makes it hard to spot:
+
+- **site-a-b** surfaced it as a *running* fault — every sync call threw, replication reported
+  `connected: false` with a backlog of 892, and the faulted session retried about once a minute.
+  The node stayed up and otherwise healthy-looking.
+- **site-a-a** was corrupt too, but **latently** — nothing on the running build ever read the
+  damaged pages. It only became fatal on the *next upgrade*, when a new build's boot-time
+  re-registration read pages the old build never touched: `SqliteLocalDb` construction threw and
+  the container **SIGABRT crash-looped** (×12).
+
+The lesson to carry: a clean-looking node proves nothing about its file. Corruption can sit latent
+until an unrelated code change happens to read further into the database.
+
+**2. The OrbStack VM idle-sleeping mid-operation.** The VM can suspend under its idle policy while
+work is in flight. `~/.orbstack/log/vmgr.log` records it plainly:
+
+```
+vmgr ... msg=sleep
+```
+
+On wake, the virtiofs layer logs repeated errors of the form:
+
+```
+refresh_nodeid: race with new lookup
+```
+
+and `docker` hangs — commands stop returning, `docker ps` comes back empty — for as long as it
+takes to notice. **The fix is `orb restart docker`**; data survives and containers auto-restart.
+Observed cost: roughly 20 minutes of a session lost to diagnosing what looked like a cluster
+failure but was a host one. If the rig goes unresponsive as a whole, check `vmgr.log` **before**
+investigating ScadaBridge.
+
+#### Recovery runbook — LocalDb corruption on a site pair (proven)
+
+This is the procedure that recovered site-a. It relies on the pair being able to reseed itself,
+so it is a *replicated* pair's recovery; an unreplicated node has no peer to reseed from and must
+be restored from a backup instead.
+
+1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same
+   stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason.
+2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping
+   them for forensics:
+
+   ```sh
+   # per affected node's data directory
+   mv site-localdb.db     site-localdb.db.corrupt-$(date +%F)
+   mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
+   mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)
+   ```
+
+   The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh
+   database is its own corruption.
+3. **Start both nodes together.** The fresh files reseed themselves from two independent paths:
+   `SiteReconciliationActor`'s node-startup fetch pulls config back from central when central
+   reports gaps, and LocalDb's **CDC snapshot resync** merges the peer's rows under LWW.
+4. **Verify convergence**, do not assume it: `LocalDbReplicationConnected` should read `true` and
+   `LocalDbOplogBacklog` should settle to ~0 on the site health report. (See *Reading the
+   replication backlog* above for why a transient spike is normal and not a fault.)
+
+On the live recovery, both quarantined files were large (394 MB and 416 MB), the pair converged to
+`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the
+digest handshake green.
+
+#### Mitigation options, with their trade-offs
+
+None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted
+position is the runbook above. Recorded so the choice is a choice:
+
+| Option | Gains | Costs |
+|--------|-------|-------|
+| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. |
+| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. |
+| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. |
+
 ### Central-Site Communication
 
 Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed