From b8f91bab2ddbe59c90033a4821377a4050e3063e Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 2 Aug 2026 01:04:55 -0400 Subject: [PATCH] =?UTF-8?q?feat(cluster):=20enable=20the=20#33=20bootstrap?= =?UTF-8?q?=20guard=20on=20the=20docker=20rig=20=E2=80=94=20live=20gate=20?= =?UTF-8?q?PASS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deploy.sh's simultaneous recreate split site pairs twice on 2026-08-01 with the guard off (mutual InitJoinNack, each node forming its own 1-node cluster; a per-pair coin flip compose depends_on does not prevent). Guard enabled on all 8 rig nodes: two consecutive simultaneous-start trials (full redeploy + full-topology compose restart) converged all four pairs deterministically — founder self-first on every lower address, peer-first join on every higher, zero splits. This closes the deferred issue-acceptance live gate; the switch stays default-off everywhere else. --- CLAUDE.md | 2 +- docker/central-node-a/appsettings.Central.json | 9 ++++++++- docker/central-node-b/appsettings.Central.json | 9 ++++++++- docker/site-a-node-a/appsettings.Site.json | 9 ++++++++- docker/site-a-node-b/appsettings.Site.json | 9 ++++++++- docker/site-b-node-a/appsettings.Site.json | 9 ++++++++- docker/site-b-node-b/appsettings.Site.json | 9 ++++++++- docker/site-c-node-a/appsettings.Site.json | 9 ++++++++- docker/site-c-node-b/appsettings.Site.json | 9 ++++++++- docs/requirements/Component-ClusterInfrastructure.md | 4 +++- 10 files changed, 68 insertions(+), 10 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 39f6df41..8c40ddc4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -177,7 +177,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - Automatic dual-node recovery from persistent storage. - **Active/standby is decided by `ActiveNodeEvaluator.SelfIsOldestUp`, never by cluster leadership** — see the Architecture note above. `/health/active` is **central-only** (site nodes map no `/health/*` at all) and backs both Traefik's active-node routing and `IActiveNodeGate`, so the proxy and the Inbound API always agree on which node is active. Central reaches a site by dialling `GrpcNodeAAddress`/`GrpcNodeBAddress` explicitly and flipping on error; whichever node answers, the site-internal `ClusterSingletonProxy` lands the work on the active node for free, so central still never needs to track which *site* node is active. - **Seed-node ordering: every node lists ITSELF first (decision 2026-07-22) — the boot-alone gap is CLOSED.** Only `seed-nodes[0]` may self-join to form a new cluster (Akka runs `FirstSeedNodeProcess` for it, `JoinSeedNodeProcess` — which can never form one — for everyone else). All 14 shipped node appsettings now lead with the node's own address, so any node can cold-start alone and become operational unattended (~5s, `seed-node-timeout`); `StartupValidator` fails the boot if the ordering is broken (compares host AND port; Akka does no DNS canonicalisation). Two nodes cold-starting together while mutually reachable converge on ONE cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class auto-down accepts. **An external self-form timer (`Cluster.Join(SelfAddress)` after a window) was implemented and REJECTED:** it sits outside the join handshake, so on a routine standby restart — where the peer is alive but the join is stalled behind removal of the node's own stale incarnation — it fires mid-join and permanently splits the pair (measured: still split after 90s). Regression tests: `SelfFirstSeedBootstrapTests`. The keep-oldest active-crash total outage was separately closed by the auto-down decision. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering. -- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start. +- **Simultaneous-cold-start split-brain guard (opt-in, Gitea #33) — the residual self-first cost, closed.** Self-first-on-both means a *truly simultaneous* cold start (shared power/hypervisor event) races `FirstSeedNodeProcess` on BOTH nodes → two 1-node clusters that never merge (in-process loopback tests converge and hide it; real parallel VM starts don't — OtOpcUa reproduced it live). Ported from OtOpcUa (`lmxopcua` `d1dac87f`): a **dark switch `ScadaBridge:Cluster:BootstrapGuard:Enabled` (default OFF, guard-off byte-identical)**. On: `BuildHocon` emits an EMPTY seed list (Akka does not auto-join) and `ClusterBootstrapCoordinator` (`IHostedService`, registered in BOTH the Central and Site composition roots) issues ONE reachability-gated `JoinSeedNodes` — the pure `ClusterBootstrapGuard` core makes the lower canonical `host:port` the **founder** (self-first, forms immediately), the higher node TCP-probes the founder up to `PartnerProbeSeconds` (25s) and joins **peer-first** if reachable else **self-first** (cold-start-alone preserved). Decided BEFORE a single join, **never re-forms mid-handshake** (that was the rejected self-form-timer's flaw); case-insensitive founder tie-break; probe timings validated `>0` when enabled. Residual accepted trade: founder dies in the probe→join window → higher node hangs unjoined, coordinator WARNs, a restart recovers it. Tests: `ClusterBootstrapGuardTests` (pure) + `ClusterBootstrapCoordinatorTests` (real-ActorSystem, incl. both-cold-start-together-form-one-cluster). **ENABLED on the docker rig (all 8 nodes) 2026-08-02** after `deploy.sh`'s simultaneous recreate split site pairs twice on 2026-08-01 with the guard off; live gate PASS — two consecutive simultaneous-start trials (a full redeploy + a full-topology `docker compose restart`), all four pairs converged (every lower-address node founded self-first, every higher node probed-then-joined peer-first), zero splits. Default remains OFF elsewhere (wonder-app-vd03, docker-env2, Host defaults). See `docs/requirements/Component-ClusterInfrastructure.md` → Simultaneous cold start. ### UI & Monitoring - Central UI: Blazor Server (ASP.NET Core + SignalR) with Bootstrap CSS. No third-party component frameworks (no Blazorise, MudBlazor, Radzen, etc.). Build custom Blazor components for tables, grids, forms, etc. diff --git a/docker/central-node-a/appsettings.Central.json b/docker/central-node-a/appsettings.Central.json index 21c34ce1..3e63042e 100644 --- a/docker/central-node-a/appsettings.Central.json +++ b/docker/central-node-a/appsettings.Central.json @@ -15,7 +15,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)." diff --git a/docker/central-node-b/appsettings.Central.json b/docker/central-node-b/appsettings.Central.json index 4fce373f..411cffff 100644 --- a/docker/central-node-b/appsettings.Central.json +++ b/docker/central-node-b/appsettings.Central.json @@ -15,7 +15,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { "_comment": "ConfigurationDb/MachineDataDb (which carry the dev SQL password) are supplied as ScadaBridge__Database__* whole-key env overrides in docker/docker-compose.yml (dev-only-insecure, mirroring the ApiKeyPepper convention). The same dev password already ships committed for the sibling scadabridge-mssql container (infra/docker-compose.yml MSSQL_SA_PASSWORD) — this is consolidation, not new exposure. Env overrides layer over JSON before the ${secret:} expander runs, so the dev cluster boots with no KEK/secret store. Real/prod config uses ${secret:} tokens in src/.../appsettings.Central.json (T4)." diff --git a/docker/site-a-node-a/appsettings.Site.json b/docker/site-a-node-a/appsettings.Site.json index ddddc6cd..47877e02 100644 --- a/docker/site-a-node-a/appsettings.Site.json +++ b/docker/site-a-node-a/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docker/site-a-node-b/appsettings.Site.json b/docker/site-a-node-b/appsettings.Site.json index 1bcb5b9a..676696a6 100644 --- a/docker/site-a-node-b/appsettings.Site.json +++ b/docker/site-a-node-b/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docker/site-b-node-a/appsettings.Site.json b/docker/site-b-node-a/appsettings.Site.json index d7270bf1..84b77600 100644 --- a/docker/site-b-node-a/appsettings.Site.json +++ b/docker/site-b-node-a/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docker/site-b-node-b/appsettings.Site.json b/docker/site-b-node-b/appsettings.Site.json index acea128d..3ca78cd1 100644 --- a/docker/site-b-node-b/appsettings.Site.json +++ b/docker/site-b-node-b/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docker/site-c-node-a/appsettings.Site.json b/docker/site-c-node-a/appsettings.Site.json index 4d0abfdd..d58e13b5 100644 --- a/docker/site-c-node-a/appsettings.Site.json +++ b/docker/site-c-node-a/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docker/site-c-node-b/appsettings.Site.json b/docker/site-c-node-b/appsettings.Site.json index f055cf5a..cc155b04 100644 --- a/docker/site-c-node-b/appsettings.Site.json +++ b/docker/site-c-node-b/appsettings.Site.json @@ -18,7 +18,14 @@ "StableAfter": "00:00:15", "HeartbeatInterval": "00:00:02", "FailureDetectionThreshold": "00:00:10", - "MinNrOfMembers": 1 + "MinNrOfMembers": 1, + "_bootstrapGuard": "Gitea #33 guard ENABLED on the docker rig (2026-08-02): deploy.sh recreates all containers simultaneously, which twice split site pairs into two 1-node clusters on 2026-08-01. Lower host:port founds self-first; the higher node TCP-probes then joins peer-first.", + "BootstrapGuard": { + "Enabled": true, + "PartnerProbeSeconds": 25, + "PartnerProbeIntervalMs": 500, + "ProbeConnectTimeoutMs": 1000 + } }, "Database": { // Migration-only as of LocalDb Phase 2. The site config tables now live in the diff --git a/docs/requirements/Component-ClusterInfrastructure.md b/docs/requirements/Component-ClusterInfrastructure.md index 35b98d83..c213729f 100644 --- a/docs/requirements/Component-ClusterInfrastructure.md +++ b/docs/requirements/Component-ClusterInfrastructure.md @@ -156,7 +156,9 @@ The **bootstrap guard** eliminates the split without giving up cold-start-alone. | `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. | | `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. | -Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); the docker rig's compose `depends_on` does that today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once. +Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); compose `depends_on` was assumed to do that on the docker rig, but it demonstrably does not — see below. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once. + +**Live status (2026-08-02): ENABLED on the primary docker rig (all 8 nodes), live gate PASS.** With the guard off, `docker/deploy.sh`'s simultaneous container recreate reproduced the split naturally TWICE on 2026-08-01 — first run split both site-b and site-c (mutual `InitJoinNack`, each node "JOINING itself and forming a new cluster"), second run split site-c while the other pairs converged: the race is a per-pair coin flip, and compose `depends_on` does not serialize far enough to prevent it. With the guard enabled on every rig node, two consecutive simultaneous-start trials (a full redeploy plus a full-topology `docker compose restart`) converged all four pairs deterministically — every lower-address node logged founder/self-first, every higher node logged partner-REACHABLE/peer-first and received `Welcome from` the founder; zero splits, `/health/ready` green. This is the issue-acceptance live gate that was deferred at implementation time. The switch stays default-off in the Host appsettings and everywhere else (wonder-app-vd03, docker-env2). ### Manual Failover (admin-triggered)