diff --git a/docs/deployment/topology-guide.md b/docs/deployment/topology-guide.md index 5320168c..8836cd8c 100644 --- a/docs/deployment/topology-guide.md +++ b/docs/deployment/topology-guide.md @@ -311,6 +311,103 @@ a rejoin is safe and self-correcting (verified live: a node stopped and restarte with both nodes byte-identical and zero duplicates). Beyond it, rebuild the returning node's database from its peer rather than letting it rejoin. +### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard + +**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a +host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and +not something a production Linux deployment is exposed to. It is documented here because it has +already cost two databases and half a session, and because the recovery is non-obvious. + +Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree +— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`, +`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the +Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety +contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true, +an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate` +— which `deploy.sh` uses — is exactly such an abrupt kill. + +Two distinct failure modes have been observed live (both 2026-08-15): + +**1. Silent database corruption.** Both site-a LocalDb files were found malformed: + +``` +SQLite Error 11: 'database disk image is malformed' +``` + +Note how differently the two nodes presented, because this is the part that makes it hard to spot: + +- **site-a-b** surfaced it as a *running* fault — every sync call threw, replication reported + `connected: false` with a backlog of 892, and the faulted session retried about once a minute. + The node stayed up and otherwise healthy-looking. +- **site-a-a** was corrupt too, but **latently** — nothing on the running build ever read the + damaged pages. It only became fatal on the *next upgrade*, when a new build's boot-time + re-registration read pages the old build never touched: `SqliteLocalDb` construction threw and + the container **SIGABRT crash-looped** (×12). + +The lesson to carry: a clean-looking node proves nothing about its file. Corruption can sit latent +until an unrelated code change happens to read further into the database. + +**2. The OrbStack VM idle-sleeping mid-operation.** The VM can suspend under its idle policy while +work is in flight. `~/.orbstack/log/vmgr.log` records it plainly: + +``` +vmgr ... msg=sleep +``` + +On wake, the virtiofs layer logs repeated errors of the form: + +``` +refresh_nodeid: race with new lookup +``` + +and `docker` hangs — commands stop returning, `docker ps` comes back empty — for as long as it +takes to notice. **The fix is `orb restart docker`**; data survives and containers auto-restart. +Observed cost: roughly 20 minutes of a session lost to diagnosing what looked like a cluster +failure but was a host one. If the rig goes unresponsive as a whole, check `vmgr.log` **before** +investigating ScadaBridge. + +#### Recovery runbook — LocalDb corruption on a site pair (proven) + +This is the procedure that recovered site-a. It relies on the pair being able to reseed itself, +so it is a *replicated* pair's recovery; an unreplicated node has no peer to reseed from and must +be restored from a backup instead. + +1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same + stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason. +2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping + them for forensics: + + ```sh + # per affected node's data directory + mv site-localdb.db site-localdb.db.corrupt-$(date +%F) + mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F) + mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F) + ``` + + The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh + database is its own corruption. +3. **Start both nodes together.** The fresh files reseed themselves from two independent paths: + `SiteReconciliationActor`'s node-startup fetch pulls config back from central when central + reports gaps, and LocalDb's **CDC snapshot resync** merges the peer's rows under LWW. +4. **Verify convergence**, do not assume it: `LocalDbReplicationConnected` should read `true` and + `LocalDbOplogBacklog` should settle to ~0 on the site health report. (See *Reading the + replication backlog* above for why a transient spike is normal and not a fault.) + +On the live recovery, both quarantined files were large (394 MB and 416 MB), the pair converged to +`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the +digest handshake green. + +#### Mitigation options, with their trade-offs + +None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted +position is the runbook above. Recorded so the choice is a choice: + +| Option | Gains | Costs | +|--------|-------|-------| +| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. | +| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. | +| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. | + ### Central-Site Communication Three transports cross the boundary, not one — **all now gRPC or HTTP; Akka ClusterClient was removed