infra(docker)+test(siteeventlog): close register tripwires 79 + 93 — rig data to named volumes; pin site_events schema to the cap-model calibration
Row 79 executed its own prescribed remedy without waiting for a second corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site nodes) move from virtiofs bind mounts to named docker volumes (scadabridge-<node>-data), seeded in place with both rigs stopped and live-verified (site-a localDbReplicationConnected=true / backlog 0 over the migrated files; env2 stable, 0 restarts). The feared inspectability loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/. The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2 now carries the x-secrets-store-env posture + central data volumes. Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests pins the site_events columns/indexes the EventLogPurgeService size model is calibrated against, failing with a pointer to the memo §4 recalibration on any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire) annotated accordingly.
This commit is contained in:
@@ -359,18 +359,25 @@ database from its peer rather than letting it rejoin.
|
||||
|
||||
### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard
|
||||
|
||||
**Status: MITIGATED 2026-08-18 (register row 79) — the rigs' data dirs moved to named docker
|
||||
volumes**, taking virtiofs out of the SQLite write path entirely. The hazard description and the
|
||||
proven recovery runbook below are retained because they explain *why* the volumes exist, and
|
||||
because failure mode 2 (the VM idle-sleep) is unaffected by the move.
|
||||
|
||||
**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a
|
||||
host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and
|
||||
not something a production Linux deployment is exposed to. It is documented here because it has
|
||||
already cost two databases and half a session, and because the recovery is non-obvious.
|
||||
|
||||
Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree
|
||||
— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`,
|
||||
`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the
|
||||
Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety
|
||||
contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true,
|
||||
an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate`
|
||||
— which `deploy.sh` uses — is exactly such an abrupt kill.
|
||||
Until 2026-08-18, every ScadaBridge data file on the rig rode a **relative bind mount** out of the
|
||||
repo working tree — `./site-a-node-a/data:/app/data` and its siblings, which is where
|
||||
`site-localdb.db`, `auditlog.db` and the secrets store all live. On macOS that path crosses
|
||||
**virtiofs** to reach the Linux VM, and **virtiofs does not give SQLite true `fsync` durability**.
|
||||
SQLite's crash-safety contract assumes a successful `fsync` means the bytes are on stable storage;
|
||||
when that is not true, an abrupt container kill mid-write can leave a torn file.
|
||||
`docker compose up --force-recreate` — which `deploy.sh` uses — is exactly such an abrupt kill.
|
||||
Today the same paths are **named volumes** (`scadabridge-<node>-data`,
|
||||
`scadabridge-env2-<node>-data`) on native ext4 inside the OrbStack VM, where `fsync` is real.
|
||||
|
||||
Two distinct failure modes have been observed live (both 2026-08-15):
|
||||
|
||||
@@ -421,13 +428,16 @@ be restored from a backup instead.
|
||||
1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same
|
||||
stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason.
|
||||
2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping
|
||||
them for forensics:
|
||||
them for forensics. The data dir now lives in the node's named volume, so run the renames
|
||||
through a throwaway container (don't write through the host-side `~/OrbStack` view):
|
||||
|
||||
```sh
|
||||
# per affected node's data directory
|
||||
mv site-localdb.db site-localdb.db.corrupt-$(date +%F)
|
||||
mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
|
||||
mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)
|
||||
# per affected node — volume name is scadabridge-<node>-data
|
||||
docker run --rm -v scadabridge-site-a-node-a-data:/d alpine sh -c '
|
||||
cd /d
|
||||
mv site-localdb.db site-localdb.db.corrupt-$(date +%F)
|
||||
mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
|
||||
mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)'
|
||||
```
|
||||
|
||||
The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh
|
||||
@@ -443,16 +453,25 @@ On the live recovery, both quarantined files were large (394 MB and 416 MB), the
|
||||
`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the
|
||||
digest handshake green.
|
||||
|
||||
#### Mitigation options, with their trade-offs
|
||||
#### Mitigation applied (2026-08-18): named docker volumes
|
||||
|
||||
None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted
|
||||
position is the runbook above. Recorded so the choice is a choice:
|
||||
The "accept + runbook" position was retired without waiting for a second corruption. All ten
|
||||
data dirs (8 `docker/` nodes + 2 `docker-env2/` site nodes) now ride named volumes
|
||||
(`scadabridge-<node>-data` / `scadabridge-env2-<node>-data`), declared in each compose file with
|
||||
explicit `name:`s and seeded from the old bind-mount dirs at migration time. What changed the
|
||||
original trade-off calculus: the assumed cost — losing host-side inspectability — turned out to
|
||||
be much smaller than documented, because OrbStack exposes every volume on the Mac at
|
||||
`~/OrbStack/docker/volumes/<name>/`, so `site-localdb.db` can still be opened from the
|
||||
host for **read-only inspection**. (Never write through that path: it reintroduces exactly the
|
||||
virtiofs layer the volumes exist to avoid.) The appsettings and logs bind mounts deliberately
|
||||
stayed: config is read-only, and Serilog's append-only log files have no fsync-durability
|
||||
requirement worth losing `tail -f` from the repo tree for.
|
||||
|
||||
| Option | Gains | Costs |
|
||||
|--------|-------|-------|
|
||||
| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. |
|
||||
| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. |
|
||||
| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. |
|
||||
Residual exposure: **failure mode 2 is availability-only and still real** — the VM can still
|
||||
idle-sleep mid-operation and hang `docker` until `orb restart docker`; data now survives that
|
||||
on ext4. Disabling OrbStack's idle pause remains an un-version-controllable host setting and was
|
||||
left alone. The old `docker/<node>/data/` dirs were left in place as a dormant pre-migration
|
||||
backup; they are no longer mounted and can be deleted once the volumes have proven themselves.
|
||||
|
||||
### Central-Site Communication
|
||||
|
||||
|
||||
Reference in New Issue
Block a user