dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132)

Browse Source 
A node frozen past failure detection + StableAfter, or both sides of a partition
that heals after each downed the other, used to stay a second active node until an
operator restarted one: the peer quarantines the removed UID and Akka.Cluster
1.5.62 never acts on ThisActorSystemQuarantinedEvent.

ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via
ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for
leader-auto-down only) watches two signals:
- S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer
  is still a live member here, only this side can be shut out -> restart now.
- S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still
  runs the incarnation this node removed. A healed partition yields only S2, on
  both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then
  tiebreak.
Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower
stays and restarts only if still split one grace later. Removed peers are probed
(Identify, 5s -> 60s back-off) since nothing else contacts them after mutual
removal. Restart = Cluster.Down(self) -> existing self-down path.

The Host now exits 75 after an unplanned ActorSystem termination (and sets it on
the Windows service before it reports stopped): SCM recovery with failureflag 1
only restarts a service that stops with a non-zero code, and the watchdog used
to exit 0. Shipped default strategy stays auto-down.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 20:57:42 -05:00
parent a633005f4f
commit 952db02aab
14 changed files with 1646 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-ClusterInfrastructure.md
+20 -2
View File
@@ -104,6 +104,24 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
Without it, a node that was **frozen** (VM pause, long GC stall, `docker pause`) for longer than failure detection plus `StableAfter` comes back as a **second, permanent active node**: its peer has already downed and removed it, quarantining its UID, and Akka.Cluster 1.5.62 never tells the frozen node — so it downs the peer in its own one-node view. Both sides of a **real partition that heals** after each side downed the other end the same way. This happens under `auto-down` too (Akka core behaviour); the in-repo provider is where it is fixed.
With `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` on (**the default whenever `leader-auto-down` is selected**), the `ShutOutRestart` actor (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs`, a child of the downing actor) restarts the stale side:
| Signal (Akka.NET 1.5.62) | Who sees it | Decision |
|---|---|---|
| **S1** `ThisActorSystemQuarantinedEvent` — the peer has quarantined *this* node — while the peer is still a live member in this node's view | the frozen node | **restart now**; no tiebreak (this node has not removed the peer, so the peer cannot be shut out by it) |
| **S1** after this node has removed the peer too | either side of a split | **tiebreak now** |
| **S2** `AssociationErrorEvent` "remote system has a UID that has been quarantined" — the peer still runs the incarnation *this* node removed | both sides of a healed partition; the survivor of a freeze | wait `ShutOutGrace` (a frozen peer restarts itself on S1 meanwhile); still split at the end ⇒ **tiebreak** |
**Tiebreak: the higher address (host, then port — Akka's `Member.AddressOrdering`) restarts; the lower stays.** The addresses are in every signal and in both nodes' seed lists, so both nodes compute the same answer without talking. Member age (up-number) was rejected: by the time a signal fires both sides have removed each other, so the peer's up-number exists only as remembered history that can differ (an unconverged join, a restart since), and a disagreement means both restart or neither. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. The lower node gives its peer one more `ShutOutGrace` and restarts only if the peer has neither rejoined nor gone quiet by then (the single-sided case where the stale side happened to be the lower address).
After both sides have removed each other neither sends the other anything, so a healed partition would stay silent: the actor **probes** (an `Identify` to `/system/cluster/core/daemon`) every peer it saw removed after a Down, backing off 5 s → 60 s, until that address is a member again. Each probe of a dead peer logs one Akka association warning.
"Restart" is the existing self-down path (below): `Cluster.Down(self)` ⇒ the `ActorSystem` terminates ⇒ the process exits **75** ⇒ the supervisor restarts it and it rejoins as standby. Every decision logs a `[leader-auto-down]` line: `shut out by <peer> … tiebreak not needed — this node RESTARTS`, `split from <peer> … Giving it <grace> …`, `Tiebreak says RESTART|STAY`, `rejoined … stays`. Settings: `RestartWhenShutOut` (bool, default `true`) and `ShutOutGrace` (default `60s`, must be positive, must cover the peer's worst-case shutdown). Off restores the pre-#132 behaviour (dual-active until an operator restarts one side). Proven by `ShutOutRestartTests` (unit) and `ShutOutRestartScenarioTests` (real two-node clusters, freeze shape and healed partition); rig evidence in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7.
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
@@ -116,8 +134,8 @@ Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `
When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits — with code 75** (`AkkaHostedService.SelfDownExitCode`, `EX_TEMPFAIL`; ScadaBridge#132). Under the Windows service lifetime the code is also set on the service (`ServiceBase.ExitCode`) before the stop, so the SCM sees it.
3. The service supervisor restarts it — docker `restart: unless-stopped` (any exit code), or Windows service recovery actions (`sc.exe failure … restart/…` **plus `sc.exe failureflag <svc> 1`**, both set by the wonder install script `deploy/wonder-app-vd03/install.ps1`, which is kept outside git). With `failureflag 1` the SCM runs the recovery actions for a service that stops with a **non-zero** exit code; before #132 the watchdog exit reported 0, which the SCM treats as a clean stop. (ScadaBridge runs as native .NET Windows services; NSSM wraps only the LDAP directory.)
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
### Seed Node Ordering