feat(cluster): simultaneous-cold-start split-brain guard (opt-in)

Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

docs/Redundancy.md

@@ -401,11 +401,61 @@ Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it
 no seed URI anywhere, so the rule would silently exempt the entire rig.
 
 Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
-seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
-nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
-`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
-**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
-`auto-down` strategy already accepts, with the same recovery (restart one side).
+seed answered, so a peer booting after the survivor formed simply joins it as the youngest member.
+
+**Simultaneous cold-start CAN split — this is the residual gap the bootstrap guard below closes.** When
+BOTH nodes start from nothing at the same instant, each runs `FirstSeedNodeProcess`, and if neither has
+formed before the other's `seed-node-timeout` expires, EACH self-joins and forms its own single-node
+cluster — two Primaries in one pair. On in-process loopback the `InitJoin` handshake usually resolves fast
+enough to converge, but the Phase 6 live gate reproduced the split reliably on the docker-dev rig (real
+container start timing + DNS): both nodes logged `JOINING itself … forming a new cluster` and both advertised
+ServiceLevel 250. Two independent clusters do **not** auto-merge, so the split persists until an operator
+restarts one side.
+
+### Bootstrap guard (`Cluster:BootstrapGuard`, opt-in)
+
+The guard eliminates the simultaneous-start split without giving up cold-start-alone. It is a **dark switch,
+default off** — a node with `Cluster:BootstrapGuard:Enabled=false` (the default) keeps Akka's config-driven
+self-first auto-join exactly as above.
+
+When enabled, the node is given **no config seed nodes** (so Akka does not auto-join), and
+`ClusterBootstrapCoordinator` picks the join order from a deterministic address tie-break plus a reachability
+probe:
+
+- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins
+  self-first and forms immediately if no peer answers — no probe, no delay.
+- The **higher** node probes its partner's Akka port (a plain TCP connect; the partner binds its port well
+  before it forms) for up to `PartnerProbeSeconds` (default 25 s): **reachable ⇒ peer-first** (it joins the
+  founder, never races it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so
+  it forms alone — cold-start-alone preserved for the higher node too).
+
+The order is decided **before** issuing a single `Cluster.JoinSeedNodes`, from an explicit reachability
+signal — it never re-decides mid-handshake, the failure mode that retired the `SelfFormAfter` watchdog.
+
+**Residual trade-off (accepted, operator-visible):** once the higher node commits peer-first it cannot
+self-form (Akka's `JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window,
+the higher node hangs unjoined; the coordinator logs a clear WARNING after a bounded grace, and a **restart
+recovers it** (the guard re-runs, finds the founder down, and forms alone). Config keys:
+
+| Key | Default | Notes |
+|---|---|---|
+| `Cluster:BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere. |
+| `Cluster:BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot. |
+| `Cluster:BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. |
+| `Cluster:BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. |
+
+On the docker-dev rig the **site-a pair is the enablement demo** (guard on, startup serialization removed so
+both nodes cold-start simultaneously); **site-b keeps the `depends_on: service_healthy` startup serialization
+with the guard off**, as the A/B control. Either mechanism prevents the split; the guard is the one that also
+works on production hardware, where compose `depends_on` does not exist.
+
+The alternative to the guard is **operational**: stagger the two VMs' service-manager start, or bring the
+designated founder VM up first — see the Phase 7 co-located operator runbook
+(`docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
+
+The residual risk that remains even with the guard is a boot-time **partition** (both cold-starting, mutually
+unreachable from the start): both form, which is the same dual-active class the `auto-down` strategy already
+accepts, with the same recovery (restart one side).
 
 Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
 in-process clusters through the production bootstrap at production failure-detection timings: a lone