From d1dac87f6fad2ed368eb5da3c49c136af02ea734 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 24 Jul 2026 08:35:16 -0400 Subject: [PATCH] docs: record the bootstrap guard (CLAUDE.md + Phase 7 note) The simultaneous-cold-start split-brain, carried as a Phase 7 "candidate follow-up, not shipped", is now closed by the opt-in Cluster:BootstrapGuard (279d1d0f). Documents the guard in CLAUDE.md's bootstrap section and marks the Phase 7 finding closed. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW --- CLAUDE.md | 2 ++ .../2026-07-24-mesh-phase7-failover-drills.md | 17 ++++++++++------- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3c8f2aea..d274a661 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -226,6 +226,8 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec **A third bootstrap gap closed 2026-07-22 — downing was never the whole story, and the fix changed twice.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka runs `FirstSeedNodeProcess` — the only bootstrap path that can form a *new* cluster when no peer answers `InitJoin` — exclusively when `seed-nodes[0]` is the node’s own address; every other node runs `JoinSeedNodeProcess` and retries `InitJoin` forever. **`Cluster:SeedNodes` is therefore ORDERED: every node that is one of its own seeds lists ITSELF first, partner second** (docker-dev `central-2` was swapped; site nodes list only `central-1` and are legitimately not seeds). `AkkaClusterOptionsValidator` (`AddValidatedOptions`/`ValidateOnStart` in `AddOtOpcUaCluster`) enforces it at boot — **conditionally** (self must be entry 0 only *if* self is in the list, or every site node would refuse to start), matching on `PublicHostname`+`Port`, never the `0.0.0.0` bind address. The earlier fix — a `Cluster:SelfFormAfter` timer calling `Cluster.Join(SelfAddress)` — is **deleted**: it sat outside Akka’s join handshake, so it could not tell "no seed answered" from "a seed answered and the join is in flight", and it islanded a manually-failed-over node live before a TCP reachability guard patched that one shape. Pinned by `SelfFirstSeedBootstrapTests` (real in-process clusters, incl. a falsifiability control proving peer-first ordering never forms) + `AkkaClusterOptionsValidatorTests`. See `docs/Redundancy.md` §"Bootstrap: self-first seed ordering". +**Simultaneous cold-start splits — the bootstrap guard closes it (opt-in, `Cluster:BootstrapGuard:Enabled`, default OFF).** Self-first on BOTH nodes means when both cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out, and forms its OWN 1-node cluster → two Primaries in one pair (the Phase 6 live gate reproduced this on docker; two separate clusters do NOT auto-merge). The guard prevents it without losing cold-start-alone: the lexicographically **lower** `host:port` node is the preferred founder (self-first, forms immediately); the **higher** node TCP-probes the partner's Akka port up to `PartnerProbeSeconds` (25s) — **reachable ⇒ peer-first (join the founder)**, **unreachable ⇒ self-first (partner dead, form alone)**. The order is chosen BEFORE the single `JoinSeedNodes`, from an explicit reachability signal — it never re-forms mid-handshake (the `SelfFormAfter` failure mode). When on, Akka gets NO config seeds (`BuildClusterOptions` empties `ClusterOptions.SeedNodes`) and `ClusterBootstrapCoordinator` drives the join. Residual trade (accepted, operator-visible via a warning + restart-recovers): the higher node hangs if the founder dies in the probe→join window. docker-dev: **site-a = guard demo** (guard on, serialization removed); **site-b = A/B control** (guard off, `depends_on: service_healthy` serialization). The guard is the production-faithful fix (compose `depends_on` doesn't exist on real VMs). Pinned by `ClusterBootstrapGuardTests` + `ClusterBootstrapCoordinatorTests` (real 2-node clusters, incl. the higher-node-cold-start-alone case). See `docs/Redundancy.md` §"Bootstrap guard". + Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file. **RESOLVED by per-cluster mesh Phase 6 (2026-07-24) — no longer per-Akka-cluster.** The "one Primary across the whole Akka mesh" limitation described above is gone; see the Phase 6 paragraph immediately below. diff --git a/docs/plans/2026-07-24-mesh-phase7-failover-drills.md b/docs/plans/2026-07-24-mesh-phase7-failover-drills.md index 0f329862..76d2805a 100644 --- a/docs/plans/2026-07-24-mesh-phase7-failover-drills.md +++ b/docs/plans/2026-07-24-mesh-phase7-failover-drills.md @@ -43,13 +43,16 @@ pair has exactly one Primary, which is what matters. - **Graceful failover restarts the node in-place and it rejoins** (no split), because a single peer that is already Up answers the restarting node's InitJoin. The **simultaneous cold-start** split (Phase 6 finding) only bites when BOTH pair nodes start from nothing at once — see the runbook mitigation. -- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs.** On the real co-located VMs a - site's two VMs can power-cycle together; two self-first seeds can then each form a 1-node cluster - (split brain, two Primaries in one pair). The docker-dev rig serializes with a founder healthcheck + - `depends_on: service_healthy`. **Production mitigation (operational, not code):** stagger the two VMs' - service-manager start, or start the designated founder VM first. A product-level guard (e.g. a brief - reachability-gated join delay on the non-founder) is a candidate follow-up if operational staggering - proves fragile — tracked as a Phase 7 note, not shipped. +- **Carried Phase-6 finding — simultaneous cold-start of both pair VMs — NOW CLOSED by a product guard + (`279d1d0f`).** On the real co-located VMs a site's two VMs can power-cycle together; two self-first + seeds can then each form a 1-node cluster (split brain, two Primaries in one pair). Two mitigations now + exist: (1) **operational** — stagger the two VMs' service-manager start / start the founder first (the + docker-dev `depends_on: service_healthy` serialization, still used by site-b); (2) **product** — the + opt-in `Cluster:BootstrapGuard` (default off), which makes the lower-address node the founder and the + higher node probe-then-join, arbitrating the race inside the join decision. Live-gated on the site-a + pair (guard on, serialization removed): simultaneous start → 250/240, no split; higher-node cold-start + -alone → self-forms → 250. See `docs/Redundancy.md` §"Bootstrap guard". The guard does NOT need the + compose `depends_on` that production hardware lacks — it is the production-faithful fix. ---