dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Cluster: guard against simultaneous-cold-start split-brain (both self-first seeds form separate 1-node clusters) #33

New Issue
Open
opened 2026-07-24 08:42:32 -04:00 by dohertj2 · 0 comments
dohertj2 commented 2026-07-24 08:42:32 -04:00
Owner
Copy Link
Copy Source

Problem — simultaneous cold-start of both pair nodes splits the cluster

ScadaBridge runs 2-node Akka clusters where both nodes are self-first seeds. Confirmed in the shipped
configs, e.g. docker/site-a-node-a/appsettings.Site.json vs docker/site-a-node-b/appsettings.Site.json:

// node-a                                             // node-b
"SeedNodes": [                                        "SeedNodes": [
  "akka.tcp://scadabridge@scadabridge-site-a-a:8082",   "akka.tcp://scadabridge@scadabridge-site-a-b:8082",
  "akka.tcp://scadabridge@scadabridge-site-a-b:8082"    "akka.tcp://scadabridge@scadabridge-site-a-a:8082"
]                                                     ]

Self-first-on-both is deliberate — it lets either node cold-start alone when its partner is dead
(only seed-nodes[0] == self runs Akka's FirstSeedNodeProcess, the sole path that can form a new
cluster). The cost: when both nodes cold-start at the same instant, each runs FirstSeedNodeProcess,
times out waiting for the other, and forms its own single-node cluster — a split brain (two oldest
nodes, two singletons, dual-active). Two independent clusters do not auto-merge, so it persists until
an operator restarts one side. This bites on any event that powers up both site VMs together (site power
restoration, hypervisor host reboot).

This is not theoretical: the sister project OtOpcUa (same ZB.MOM.WW.* Akka topology, whose cluster
shape mirrors ScadaBridge's) reproduced it reliably on its docker rig — both nodes logged
JOINING itself … forming a new cluster and both advertised the primary role. In-process loopback tests
usually converge (the InitJoin handshake resolves before the self-join deadline), which is why this
hides until a real deployment starts both containers/VMs truly in parallel.

Fix — the bootstrap guard OtOpcUa built (please port)

OtOpcUa shipped an opt-in guard that eliminates the split without giving up cold-start-alone. Design:

  • Dark switch, default OFF (Cluster:BootstrapGuard:Enabled) — off = today's config-driven self-first
    auto-join, byte-identical. On = the node is given no config seed nodes (so Akka does not auto-join)
    and a coordinator picks the join order after a reachability probe.
  • The node with the lexicographically lower canonical host:port is the preferred founder:
    self-first, forms immediately.
  • The higher node TCP-probes its partner's Akka port (a node binds its port well before it forms) up
    to PartnerProbeSeconds: reachable ⇒ peer-first (join the founder, never race it);
    unreachable after the window ⇒ self-first (partner is genuinely down — cold-start-alone preserved
    for the higher node too).
  • The order is decided before issuing a single Cluster.JoinSeedNodes, from an explicit reachability
    signal. It never re-forms mid-handshake — that was the failure mode of a prior timer-based
    SelfFormAfter approach (a Cluster.Join(self) on a bare timeout is not ignored mid-handshake, it
    wins, and it islanded a node a failover had just bounced).
  • Residual trade-off (accepted, operator-visible): once the higher node commits peer-first it cannot
    self-form (JoinSeedNodeProcess retries forever). If the founder dies in the small probe→join window,
    the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a restart
    recovers it (the guard re-runs, finds the founder down, self-forms). We deliberately do not re-decide
    mid-handshake.

Reference implementation (OtOpcUa)

  • Commit d1dac87f (repo dohertj2/lmxopcua).
  • src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapGuard.cs — pure tie-break + reachability decision
    (fully unit-tested).
  • src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapCoordinator.cs — IHostedService: background probe
    • the single JoinSeedNodes + the operator-visible-hang warning.
  • AkkaClusterOptions gained ClusterBootstrapGuardOptions (Enabled, PartnerProbeSeconds [25],
    PartnerProbeIntervalMs [500], ProbeConnectTimeoutMs [1000]); validated fail-closed at boot.
  • BuildClusterOptions empties Akka's seed list when the guard is on; the coordinator is registered as a
    hosted service after AddAkka.
  • Docs: docs/Redundancy.md §"Bootstrap guard".

Review notes worth carrying over (OtOpcUa code review caught these)

  1. Make the founder tie-break case-insensitive (OrdinalIgnoreCase on host:port) — a hostname
    casing mismatch between the two nodes' configs would otherwise make both think they are the founder and
    reopen the split.
  2. Fail-fast validation of the probe timings — a PartnerProbeSeconds: 0 typo silently degrades the
    guard to "never wait, always conclude the partner is dead."
  3. Cover the higher-node-cold-start-alone case with a real-ActorSystem test — it would hang forever if
    the peer-first logic were wrong, and pure unit tests do not reproduce the true-simultaneous race.

Acceptance

  • Opt-in dark switch; guard-off behavior unchanged.
  • Live gate: bring a pair up with no startup serialization (both start simultaneously) + guard on →
    one founder, one joiner, exactly one oldest/Primary, no split; and a node cold-starting alone (partner
    down) still forms its cluster.

Notes

  • The alternative is purely operational (stagger the two VMs' service-manager start / start the founder
    first). ScadaBridge's docker rig uses compose depends_on for this today, but that does not exist on the
    real co-located VMs — the guard is the production-faithful fix.
  • OtOpcUa and ScadaBridge co-locate on the same site VMs, so a shared power event hits both products'
    pairs at once; implementing the guard in both keeps their failure/recovery model identical.
## Problem — simultaneous cold-start of both pair nodes splits the cluster ScadaBridge runs 2-node Akka clusters where **both** nodes are self-first seeds. Confirmed in the shipped configs, e.g. `docker/site-a-node-a/appsettings.Site.json` vs `docker/site-a-node-b/appsettings.Site.json`: ```jsonc // node-a // node-b "SeedNodes": [ "SeedNodes": [ "akka.tcp://scadabridge@scadabridge-site-a-a:8082", "akka.tcp://scadabridge@scadabridge-site-a-b:8082", "akka.tcp://scadabridge@scadabridge-site-a-b:8082" "akka.tcp://scadabridge@scadabridge-site-a-a:8082" ] ] ``` Self-first-on-both is deliberate — it lets **either** node cold-start alone when its partner is dead (only `seed-nodes[0] == self` runs Akka's `FirstSeedNodeProcess`, the sole path that can form a new cluster). The cost: when **both** nodes cold-start at the same instant, each runs `FirstSeedNodeProcess`, times out waiting for the other, and forms its **own single-node cluster** — a split brain (two oldest nodes, two singletons, dual-active). Two independent clusters do **not** auto-merge, so it persists until an operator restarts one side. This bites on any event that powers up both site VMs together (site power restoration, hypervisor host reboot). This is not theoretical: the sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology, whose cluster shape mirrors ScadaBridge's) reproduced it reliably on its docker rig — both nodes logged `JOINING itself … forming a new cluster` and both advertised the primary role. In-process loopback tests usually converge (the `InitJoin` handshake resolves before the self-join deadline), which is why this hides until a real deployment starts both containers/VMs truly in parallel. ## Fix — the bootstrap guard OtOpcUa built (please port) OtOpcUa shipped an opt-in guard that eliminates the split without giving up cold-start-alone. Design: - **Dark switch, default OFF** (`Cluster:BootstrapGuard:Enabled`) — off = today's config-driven self-first auto-join, byte-identical. On = the node is given **no** config seed nodes (so Akka does not auto-join) and a coordinator picks the join order after a reachability probe. - The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: self-first, forms immediately. - The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (partner is genuinely down — cold-start-alone preserved for the higher node too). - The order is decided **before** issuing a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. It **never re-forms mid-handshake** — that was the failure mode of a prior timer-based `SelfFormAfter` approach (a `Cluster.Join(self)` on a bare timeout is not ignored mid-handshake, it wins, and it islanded a node a failover had just bounced). - **Residual trade-off (accepted, operator-visible):** once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, self-forms). We deliberately do not re-decide mid-handshake. ## Reference implementation (OtOpcUa) - Commit `d1dac87f` (repo `dohertj2/lmxopcua`). - `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapGuard.cs` — pure tie-break + reachability decision (fully unit-tested). - `src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapCoordinator.cs` — `IHostedService`: background probe + the single `JoinSeedNodes` + the operator-visible-hang warning. - `AkkaClusterOptions` gained `ClusterBootstrapGuardOptions` (`Enabled`, `PartnerProbeSeconds` [25], `PartnerProbeIntervalMs` [500], `ProbeConnectTimeoutMs` [1000]); validated fail-closed at boot. - `BuildClusterOptions` empties Akka's seed list when the guard is on; the coordinator is registered as a hosted service after `AddAkka`. - Docs: `docs/Redundancy.md` §"Bootstrap guard". ## Review notes worth carrying over (OtOpcUa code review caught these) 1. Make the founder tie-break **case-insensitive** (`OrdinalIgnoreCase` on `host:port`) — a hostname casing mismatch between the two nodes' configs would otherwise make both think they are the founder and reopen the split. 2. **Fail-fast validation** of the probe timings — a `PartnerProbeSeconds: 0` typo silently degrades the guard to "never wait, always conclude the partner is dead." 3. Cover the **higher-node-cold-start-alone** case with a real-ActorSystem test — it would hang forever if the peer-first logic were wrong, and pure unit tests do not reproduce the true-simultaneous race. ## Acceptance - Opt-in dark switch; guard-off behavior unchanged. - Live gate: bring a pair up with **no** startup serialization (both start simultaneously) + guard on → one founder, one joiner, exactly one oldest/Primary, no split; and a node cold-starting alone (partner down) still forms its cluster. ## Notes - The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first). ScadaBridge's docker rig uses compose `depends_on` for this today, but that does not exist on the real co-located VMs — the guard is the production-faithful fix. - OtOpcUa and ScadaBridge co-locate on the same site VMs, so a shared power event hits **both** products' pairs at once; implementing the guard in both keeps their failure/recovery model identical.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/ScadaBridge#33
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?