infra(docker)+test(siteeventlog): close register tripwires 79 + 93 — rig data to named volumes; pin site_events schema to the cap-model calibration

Row 79 executed its own prescribed remedy without waiting for a second
corruption: all ten rig data dirs (8 docker/ nodes + 2 docker-env2/ site
nodes) move from virtiofs bind mounts to named docker volumes
(scadabridge-<node>-data), seeded in place with both rigs stopped and
live-verified (site-a localDbReplicationConnected=true / backlog 0 over
the migrated files; env2 stable, 0 restarts). The feared inspectability
loss is moot — OrbStack exposes volumes at ~/OrbStack/docker/volumes/.
The rebuild surfaced latent env2 drift (pre-Secrets-0.6.2 image): env2
now carries the x-secrets-store-env posture + central data volumes.

Row 93's silent maintenance trap is now loud: SiteEventsSchemaCalibrationTests
pins the site_events columns/indexes the EventLogPurgeService size model is
calibrated against, failing with a pointer to the memo §4 recalibration on
any schema drift. Rows 92 (accepted by design) and 94 (standing tripwire)
annotated accordingly.
This commit is contained in:
Joseph Doherty
2026-08-18 08:43:52 -04:00
parent 5936573e71
commit 324ba369d9
8 changed files with 284 additions and 83 deletions
+39 -20
View File
@@ -359,18 +359,25 @@ database from its peer rather than letting it rejoin.
### Docker-rig data files on macOS/OrbStack — a virtiofs corruption + availability hazard
**Status: MITIGATED 2026-08-18 (register row 79) — the rigs' data dirs moved to named docker
volumes**, taking virtiofs out of the SQLite write path entirely. The hazard description and the
proven recovery runbook below are retained because they explain *why* the volumes exist, and
because failure mode 2 (the VM idle-sleep) is unaffected by the move.
**Scope: the local `docker/` and `docker-env2/` development rigs on macOS only.** This is a
host-level property of the filesystem the rig's data files sit on, **not a ScadaBridge defect** and
not something a production Linux deployment is exposed to. It is documented here because it has
already cost two databases and half a session, and because the recovery is non-obvious.
Every ScadaBridge data file on the rig rides a **relative bind mount** out of the repo working tree
— `./site-a-node-a/data:/app/data` and its siblings, which is where `site-localdb.db`,
`auditlog.db` and the secrets store all live. On macOS that path crosses **virtiofs** to reach the
Linux VM, and **virtiofs does not give SQLite true `fsync` durability**. SQLite's crash-safety
contract assumes a successful `fsync` means the bytes are on stable storage; when that is not true,
an abrupt container kill mid-write can leave a torn file. `docker compose up --force-recreate`
— which `deploy.sh` uses — is exactly such an abrupt kill.
Until 2026-08-18, every ScadaBridge data file on the rig rode a **relative bind mount** out of the
repo working tree — `./site-a-node-a/data:/app/data` and its siblings, which is where
`site-localdb.db`, `auditlog.db` and the secrets store all live. On macOS that path crosses
**virtiofs** to reach the Linux VM, and **virtiofs does not give SQLite true `fsync` durability**.
SQLite's crash-safety contract assumes a successful `fsync` means the bytes are on stable storage;
when that is not true, an abrupt container kill mid-write can leave a torn file.
`docker compose up --force-recreate` — which `deploy.sh` uses — is exactly such an abrupt kill.
Today the same paths are **named volumes** (`scadabridge-<node>-data`,
`scadabridge-env2-<node>-data`) on native ext4 inside the OrbStack VM, where `fsync` is real.
Two distinct failure modes have been observed live (both 2026-08-15):
@@ -421,13 +428,16 @@ be restored from a backup instead.
1. **Stop BOTH nodes of the site pair, together.** Not one at a time — this is the same
stop-and-start-together rule as *Site Pair Upgrades* above, for the same reason.
2. **Quarantine the database and its sidecars — do not delete them.** Rename all three, keeping
them for forensics:
them for forensics. The data dir now lives in the node's named volume, so run the renames
through a throwaway container (don't write through the host-side `~/OrbStack` view):
```sh
# per affected node's data directory
mv site-localdb.db site-localdb.db.corrupt-$(date +%F)
mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)
# per affected node — volume name is scadabridge-<node>-data
docker run --rm -v scadabridge-site-a-node-a-data:/d alpine sh -c '
cd /d
mv site-localdb.db site-localdb.db.corrupt-$(date +%F)
mv site-localdb.db-wal site-localdb.db-wal.corrupt-$(date +%F)
mv site-localdb.db-shm site-localdb.db-shm.corrupt-$(date +%F)'
```
The `-wal` and `-shm` files **must** move with the main file. Leaving a stale WAL beside a fresh
@@ -443,16 +453,25 @@ On the live recovery, both quarantined files were large (394 MB and 416 MB), the
`connected: true` / backlog 0, and a schema change in flight at the time additionally proved the
digest handshake green.
#### Mitigation options, with their trade-offs
#### Mitigation applied (2026-08-18): named docker volumes
None of these is applied today; the rig deliberately still runs on bind mounts, and the accepted
position is the runbook above. Recorded so the choice is a choice:
The "accept + runbook" position was retired without waiting for a second corruption. All ten
data dirs (8 `docker/` nodes + 2 `docker-env2/` site nodes) now ride named volumes
(`scadabridge-<node>-data` / `scadabridge-env2-<node>-data`), declared in each compose file with
explicit `name:`s and seeded from the old bind-mount dirs at migration time. What changed the
original trade-off calculus: the assumed cost — losing host-side inspectability — turned out to
be much smaller than documented, because OrbStack exposes every volume on the Mac at
`~/OrbStack/docker/volumes/<name>/`, so `site-localdb.db` can still be opened from the
host for **read-only inspection**. (Never write through that path: it reintroduces exactly the
virtiofs layer the volumes exist to avoid.) The appsettings and logs bind mounts deliberately
stayed: config is read-only, and Serilog's append-only log files have no fsync-durability
requirement worth losing `tail -f` from the repo tree for.
| Option | Gains | Costs |
|--------|-------|-------|
| Move rig data dirs to **named docker volumes** | Files live on native ext4 inside the VM, out of virtiofs entirely — removes the corruption mechanism at the source | Loses the host-side inspectability the debugging sessions lean on constantly: no more opening `site-localdb.db` from the Mac, no `ls` of a node's data dir from the repo tree. Requires a compose change. |
| **Disable OrbStack's idle pause** | Removes failure mode 2 (the mid-operation sleep and the virtiofs `refresh_nodeid` storm) | A host-level setting outside this repo, so it cannot be version-controlled or applied by `deploy.sh` — it silently does not exist on a new machine. Does nothing about failure mode 1. |
| **Accept, and use this runbook** (current) | No changes to compose or host config; full host-side inspectability retained | Corruption recurs on the next unlucky `--force-recreate`, and can sit latent until a later upgrade turns it into a crash loop. |
Residual exposure: **failure mode 2 is availability-only and still real** — the VM can still
idle-sleep mid-operation and hang `docker` until `orb restart docker`; data now survives that
on ext4. Disabling OrbStack's idle pause remains an un-version-controllable host setting and was
left alone. The old `docker/<node>/data/` dirs were left in place as a dormant pre-migration
backup; they are no longer mounted and can be deleted once the volumes have proven themselves.
### Central-Site Communication