dohertj2/lmxopcua
1
0
Fork 0
Code Issues 10 Pull Requests 1 Actions 1 Packages Projects Releases Wiki Activity

feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
v2-ci / build (push) Successful in 4m46s
Details
v2-ci / unit-tests (push) Failing after 16m9s
Details

Browse Source 
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 08:34:29 -04:00
parent c50ebcf7dc
commit 279d1d0fb1
11 changed files with 884 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs
+10
View File
@@ -390,6 +390,16 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
}
});
// Simultaneous-cold-start split-brain guard (dark switch Cluster:BootstrapGuard:Enabled, default off).
// Registered AFTER AddAkka so its StartAsync runs once Akka's hosted service has built (and, when the
// guard is off, already auto-joined) the ActorSystem. When on, the node started with no config seeds
// (BuildClusterOptions) and this coordinator issues the single reachability-gated JoinSeedNodes. It
// no-ops when the guard is off, so registering it unconditionally is safe.
builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<ActorSystem>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor