dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash

Browse Source 
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.

Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.

Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.

The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.

Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.

Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:32:42 -04:00
parent c6fb8bb0ad
commit 2964361a6f
5 changed files with 427 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Redundancy.md
+78 -39
View File
@@ -169,55 +169,94 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
## Split-brain
## Split-brain / downing
The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
express it).
**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
into the `akka.cluster.downing-provider-class` actually installed:
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
this is inherent to any 2-node cluster:
| Node lost | What keep-oldest does | Recovery |
| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
|---|---|---|
| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
sister project runs the same pattern):
Any other value **fails the host at startup** rather than falling through to a default.
### Why keep-oldest is no longer the default
The previous documentation in this section — and the code comments behind it — asserted that
`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
**That was wrong, and the error was not conservative: it described a total-outage configuration as the
recommended one.**
In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
for the upstream decision record, which OtOpcUa follows.
The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
a shared lease store both nodes can reach.
### The accepted trade
Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
downing quarantines the association.
This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
a crash under `keep-oldest` was unrecoverable without operator action anyway.
Deployments that would rather take an outage than ever run dual-active can set
`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
two-node pair that choice means *any* crash of the oldest node is a full outage.
### Recovery still depends on supervision
1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
`sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
`restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
`IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
idling forever with a dead actor system.
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
`restart: unless-stopped`.
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
(it is never the one downed); the **downed** node still does.
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
this one.
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
> partition and a process death in mind when reading it.
There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
or a lightweight witness) — a deliberate future option, not the current 2-node posture.
`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
losing one still looks correct in the file.
There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do
automatically.
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Primary data-plane gate (writes, acks, alerts emit)