feat(cluster): simultaneous-cold-start split-brain guard (opt-in)

Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docker-dev/docker-compose.yml

@@ -516,6 +516,12 @@ services:
       # FetchAndCache, never gossip; central and site-a share no seed list.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
       Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
+      # Simultaneous-cold-start split-brain guard (SITE-A enablement demo). With the pair no longer
+      # startup-serialized, this is what stops both self-first seeds forming their own 1-node cluster:
+      # the lower-address node (site-a-1) founds immediately; the higher (site-a-2) probes site-a-1 and
+      # joins it when reachable, or forms alone only if site-a-1 is truly down. Akka gets NO config
+      # seeds when this is on (BuildClusterOptions); ClusterBootstrapCoordinator issues JoinSeedNodes.
+      Cluster__BootstrapGuard__Enabled: "true"
       Cluster__Roles__0: "driver"
       Cluster__Roles__1: "cluster-SITE-A"
       # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
@@ -588,13 +594,11 @@ services:
       sql: { condition: service_healthy }
       central-1: { condition: service_started }
       migrator: { condition: service_completed_successfully }
-      # Serialize the SITE-A pair's startup so site-a-1 forms the mesh first and site-a-2 JOINS it —
-      # exactly as central-2 waits on central-1. Both nodes are self-first seeds (seed[0]=self), so BOTH
-      # run Akka's FirstSeedNodeProcess and, if they start simultaneously, EACH forms its own single-node
-      # cluster (split brain — observed in the Phase 6 live gate: both logged "JOINING itself ... forming
-      # a new cluster", both elected themselves Primary → 250/250 instead of 250/240). Waiting on site-a-1
-      # gives it the head start to be Up before site-a-2's InitJoin, so site-a-2 joins the existing mesh.
-      site-a-1: { condition: service_healthy }
+      # SITE-A is the BOOTSTRAP-GUARD enablement demo: NO startup serialization on the pair — both
+      # nodes start simultaneously (the exact race that split-brained in the Phase 6 gate). The
+      # Cluster:BootstrapGuard (enabled in both site-a env blocks) is the ONLY thing preventing the
+      # split: it makes the lower-address node the founder and the higher-address node probe-then-join.
+      # site-b keeps the depends_on:service_healthy serialization + guard OFF, as the A/B control.
     environment:
       OTOPCUA_ROLES: "driver,cluster-SITE-A"
       # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
@@ -617,6 +621,9 @@ services:
       # second) — it no longer seeds off central-1.
       Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053"
       Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053"
+      # Bootstrap-guard enablement demo — see site-a-1. site-a-2 is the HIGHER address, so the guard
+      # makes it probe site-a-1 and join it (peer-first) rather than race-form its own cluster.
+      Cluster__BootstrapGuard__Enabled: "true"
       Cluster__Roles__0: "driver"
       Cluster__Roles__1: "cluster-SITE-A"
       # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see