dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(cluster): enable the #33 bootstrap guard on the docker rig — live gate PASS

Browse Source 
deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with
the guard off (mutual InitJoinNack, each node forming its own 1-node
cluster; a per-pair coin flip compose depends_on does not prevent). Guard
enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full
redeploy + full-topology compose restart) converged all four pairs
deterministically — founder self-first on every lower address, peer-first
join on every higher, zero splits. This closes the deferred issue-acceptance
live gate; the switch stays default-off everywhere else.
This commit is contained in:
Joseph Doherty
2026-08-02 01:04:55 -04:00
parent 3c9b101dfc
commit b8f91bab2d
10 changed files with 68 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-ClusterInfrastructure.md
+3 -1
View File
@@ -156,7 +156,9 @@ The **bootstrap guard** eliminates the split without giving up cold-start-alone.
| `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
| `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); compose `depends_on` was assumed to do that on the docker rig, but it demonstrably does not — see below. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
**Live status (2026-08-02): ENABLED on the primary docker rig (all 8 nodes), live gate PASS.** With the guard off, `docker/deploy.sh`'s simultaneous container recreate reproduced the split naturally TWICE on 2026-08-01 — first run split both site-b and site-c (mutual `InitJoinNack`, each node "JOINING itself and forming a new cluster"), second run split site-c while the other pairs converged: the race is a per-pair coin flip, and compose `depends_on` does not serialize far enough to prevent it. With the guard enabled on every rig node, two consecutive simultaneous-start trials (a full redeploy plus a full-topology `docker compose restart`) converged all four pairs deterministically — every lower-address node logged founder/self-first, every higher node logged partner-REACHABLE/peer-first and received `Welcome from` the founder; zero splits, `/health/ready` green. This is the issue-acceptance live gate that was deferred at implementation time. The switch stays default-off in the Host appsettings and everywhere else (wonder-app-vd03, docker-env2).
### Manual Failover (admin-triggered)