docs(cluster): document the #33 bootstrap guard; disabled BootstrapGuard block in Host appsettings

Component-ClusterInfrastructure.md gains a 'Simultaneous cold start — the
bootstrap guard' subsection (dark switch, founder/probe semantics, config table,
accepted trade) plus a forward reference from Dual-Node Recovery. Host
appsettings.Central/Site.json carry a disabled BootstrapGuard block with a
_bootstrapGuard note for operator discoverability. CLAUDE.md Cluster & Failover
note added.

CLAUDE.md

@@ -220,6 +220,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
 - Automatic dual-node recovery from persistent storage.
 - **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central never needs to know which *site* node is active: ClusterClient contact rotation reaches either receptionist and the site-internal `ClusterSingletonProxy` lands the work on the active node for free. The **exception is gRPC**, which picks `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flips on error.
 - **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering.
+- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start.
 
 ### UI & Monitoring
 - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc.

docs/requirements/Component-ClusterInfrastructure.md

@@ -137,6 +137,27 @@ Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters bui
 
 The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
 
+### Simultaneous cold start — the bootstrap guard (opt-in, Gitea #33)
+
+Self-first-on-both has a residual cost. When **both** nodes cold-start at the same instant, each is `seed-nodes[0]` for itself, so **each runs `FirstSeedNodeProcess`**, times out waiting for the other, and forms its **own** single-node cluster — a split brain (two oldest members, two singletons, dual-active) that does **not** auto-merge and persists until an operator restarts one side. In-process loopback tests usually converge (the `InitJoin` handshake resolves before either self-join deadline, so row 3 above passes), which is exactly why the split hides until a real deployment powers up both VMs truly in parallel — a shared power event or a hypervisor host reboot. The sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology) reproduced it reliably on its docker rig.
+
+The **bootstrap guard** eliminates the split without giving up cold-start-alone. It is an **opt-in dark switch** — `ScadaBridge:Cluster:BootstrapGuard:Enabled`, **default off**, so a node with the guard disabled keeps Akka's config-driven self-first auto-join byte-identical. When enabled, the node starts with **no config seed nodes** (`AkkaHostedService.BuildHocon` emits an empty seed list, so Akka does not auto-join) and a coordinator (`ClusterBootstrapCoordinator`, an `IHostedService` registered in both the Central and Site composition roots) picks the join order after a reachability probe:
+
+- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins self-first and forms immediately if no peer answers — no probe, no delay. The tie-break is **case-insensitive** (a hostname-casing mismatch between the two configs must never make both think they are the founder and re-open the split).
+- The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so cold-start-alone is preserved for the higher node too).
+- The order is decided **before** a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. The coordinator **never re-forms mid-handshake** — that was the failure mode of the rejected self-form timer above (a `Join(self)` on a bare timeout is not ignored mid-handshake; it wins and islands a node).
+
+**Residual trade-off (accepted, operator-visible).** Once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, and forms alone). The guard deliberately does not re-decide mid-handshake.
+
+| Key | Default | Meaning |
+|---|---|---|
+| `BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere (single-node install, self absent, 3+ seeds). |
+| `BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot when the guard is on. |
+| `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
+| `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |
+
+Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.
+
 ### Manual Failover (admin-triggered)
 
 An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
@@ -191,7 +212,7 @@ These values balance failover speed with stability — fast enough that data col
 
 If both nodes in a cluster fail simultaneously (e.g., site power outage):
 
-1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
+1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster (self-first ordering) and the second node joins when it starts. If both power up truly simultaneously, each self-first seed can run `FirstSeedNodeProcess` and form its own 1-node cluster — a split brain; enable the opt-in **bootstrap guard** (Gitea #33, see *Simultaneous cold start* above) on co-located VM pairs that share a power domain to make them converge on one cluster instead.
 2. **State recovery** (each node has its own local copy of all required data):
    - **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
    - **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Central.json

@@ -17,7 +17,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
+      "BootstrapGuard": {
+        "Enabled": false,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "_secrets": "Host-003: Secrets are NOT committed in this file. The ${secret:...} references below are resolved at startup by the pre-host secrets expander (before StartupValidator runs) from the encrypted secrets store (SQLite, seeded via the ZB.MOM.WW.Secrets store/CLI/UI). The KEK is supplied out-of-band via the ZB_SECRETS_MASTER_KEY environment variable and never committed; an unseeded reference fails closed (SecretNotFoundException) before any SQL/LDAP/cluster wiring. ROLLBACK: the whole-key environment override still wins — set ScadaBridge__Database__ConfigurationDb / ScadaBridge__Security__Ldap__ServiceAccountPassword / ScadaBridge__Security__JwtSigningKey and .AddEnvironmentVariables() overlays the concrete value before expansion runs, so the ${secret:...} token is never evaluated. NOTE (Task 1.4): the LDAP settings moved into the nested Security:Ldap sub-section (bound to the shared ZB.MOM.WW.Auth LdapOptions) — the service-account-password env var is ScadaBridge__Security__Ldap__ServiceAccountPassword (was ScadaBridge__Security__LdapServiceAccountPassword).",
     "Database": {

src/ZB.MOM.WW.ScadaBridge.Host/appsettings.Site.json

@@ -20,7 +20,14 @@
       "StableAfter": "00:00:15",
       "HeartbeatInterval": "00:00:02",
       "FailureDetectionThreshold": "00:00:10",
-      "MinNrOfMembers": 1
+      "MinNrOfMembers": 1,
+      "_bootstrapGuard": "Gitea #33 simultaneous-cold-start split-brain guard. Dark switch, default off (Akka's config-driven self-first auto-join, unchanged). Enable ONLY where both pair VMs can power up together (shared power/hypervisor event) with no start-order serialization: then the lower host:port founds self-first and the higher probes-then-joins, so the pair converges to one cluster instead of two 1-node clusters. Timings validated > 0 when enabled.",
+      "BootstrapGuard": {
+        "Enabled": false,
+        "PartnerProbeSeconds": 25,
+        "PartnerProbeIntervalMs": 500,
+        "ProbeConnectTimeoutMs": 1000
+      }
     },
     "Database": {
       // Migration-only as of LocalDb Phase 2. The site config tables now live in the