fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132)
A node frozen past failure detection + StableAfter, or both sides of a partition that heals after each downed the other, used to stay a second active node until an operator restarted one: the peer quarantines the removed UID and Akka.Cluster 1.5.62 never acts on ThisActorSystemQuarantinedEvent. ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for leader-auto-down only) watches two signals: - S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer is still a live member here, only this side can be shut out -> restart now. - S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still runs the incarnation this node removed. A healed partition yields only S2, on both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then tiebreak. Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower stays and restarts only if still split one grace later. Removed peers are probed (Identify, 5s -> 60s back-off) since nothing else contacts them after mutual removal. Restart = Cluster.Down(self) -> existing self-down path. The Host now exits 75 after an unplanned ActorSystem termination (and sets it on the Windows service before it reports stopped): SCM recovery with failureflag 1 only restarts a service that stops with a non-zero code, and the watchdog used to exit 0. Shipped default strategy stays auto-down. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -111,7 +111,9 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
### Downed-node recovery
|
||||
|
||||
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
|
||||
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits **with code 75** (`AkkaHostedService.SelfDownExitCode`; set by `Program` after the host stops, and on the Windows service itself before it reports `SERVICE_STOPPED`) and the service supervisor (docker `restart: unless-stopped`, Windows service recovery with `failureflag 1`, which only acts on a non-zero exit code) restarts it as a fresh incarnation (`AkkaHostedService.GetOrCreateActorSystem`).
|
||||
|
||||
Under `leader-auto-down` the same path restarts a node that its peer has **shut out** — downed, removed and quarantined it (a frozen node coming back), or both sides of a healed partition, where the higher address restarts. See `Component-ClusterInfrastructure.md` → *Shut-out restart* (ScadaBridge#132).
|
||||
|
||||
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
|
||||
|
||||
|
||||
@@ -104,3 +104,36 @@ Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the prove
|
||||
- **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that
|
||||
both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged).
|
||||
- **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs.
|
||||
|
||||
## 7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132)
|
||||
|
||||
The #116 rig drill found a gap that `AutoDowning` and its port share: a node frozen past failure detection plus the window
|
||||
comes back as a **permanent second active node**, and the same happens to both sides of a partition that heals after each
|
||||
side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote
|
||||
publishes `ThisActorSystemQuarantinedEvent` on the stale node, and Akka.Cluster never subscribes to it.
|
||||
|
||||
**Owner decision (#132):** under `leader-auto-down`, a node that is shut out restarts itself; when both sides are shut out,
|
||||
exactly one restarts, by a deterministic tiebreak. Implemented as `ShutOutRestart` (Host, a child of the downing actor, so
|
||||
`LeaderAutoDownBase` stays a line-for-line port), behind `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default
|
||||
`true` for this strategy only) and `ShutOutGrace` (default 60 s). The shipped default strategy stays `auto-down`.
|
||||
|
||||
- **Two signals, confirmed in-process.** S1 `ThisActorSystemQuarantinedEvent` (the frozen node). S2 an `AssociationErrorEvent`
|
||||
whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed
|
||||
partition produces **only S2, on both sides** — each side aborts the other's handshake with its own refused UID before a
|
||||
quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with
|
||||
`Identify` (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it.
|
||||
- **Decision.** S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this
|
||||
node removed the peer: tiebreak. S2: wait `ShutOutGrace` (a frozen peer restarts itself meanwhile), then tiebreak if still
|
||||
split. Tiebreak: **higher address restarts**, the lower stays and restarts only if the peer is still split from it one
|
||||
more grace later.
|
||||
- **Why address, not age:** see `Component-ClusterInfrastructure.md` → *Shut-out restart*. In short, both nodes hold both
|
||||
addresses always; the peer's up-number is history that can differ after the split.
|
||||
- **Exit code.** The restart is the existing self-down path, and it exposed a latent gap: the watchdog's `StopApplication()`
|
||||
exited 0, and Windows service recovery with `failureflag 1` acts only on a non-zero exit. The Host now exits 75 after an
|
||||
unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code.
|
||||
- **Contract change for #116 gate 4:** the partition still produces dual-active *while it lasts*; after it heals, one side
|
||||
(the higher address) restarts and the pair converges with no operator action.
|
||||
`LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds (it never heals).
|
||||
- **Tests:** `ShutOutRestartTests` (unit), `ShutOutRestartScenarioTests` (two real nodes: freeze shape both address orders,
|
||||
healed partition, switch off), `UnexpectedTerminationTests`, `HoconBuilderTests`, `ClusterOptionsValidatorTests`.
|
||||
Rig evidence: `2026-11-12-leader-auto-down-rig-drill.md` §7.
|
||||
|
||||
@@ -104,6 +104,24 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
|
||||
|
||||
Without it, a node that was **frozen** (VM pause, long GC stall, `docker pause`) for longer than failure detection plus `StableAfter` comes back as a **second, permanent active node**: its peer has already downed and removed it, quarantining its UID, and Akka.Cluster 1.5.62 never tells the frozen node — so it downs the peer in its own one-node view. Both sides of a **real partition that heals** after each side downed the other end the same way. This happens under `auto-down` too (Akka core behaviour); the in-repo provider is where it is fixed.
|
||||
|
||||
With `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` on (**the default whenever `leader-auto-down` is selected**), the `ShutOutRestart` actor (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs`, a child of the downing actor) restarts the stale side:
|
||||
|
||||
| Signal (Akka.NET 1.5.62) | Who sees it | Decision |
|
||||
|---|---|---|
|
||||
| **S1** `ThisActorSystemQuarantinedEvent` — the peer has quarantined *this* node — while the peer is still a live member in this node's view | the frozen node | **restart now**; no tiebreak (this node has not removed the peer, so the peer cannot be shut out by it) |
|
||||
| **S1** after this node has removed the peer too | either side of a split | **tiebreak now** |
|
||||
| **S2** `AssociationErrorEvent` "remote system has a UID that has been quarantined" — the peer still runs the incarnation *this* node removed | both sides of a healed partition; the survivor of a freeze | wait `ShutOutGrace` (a frozen peer restarts itself on S1 meanwhile); still split at the end ⇒ **tiebreak** |
|
||||
|
||||
**Tiebreak: the higher address (host, then port — Akka's `Member.AddressOrdering`) restarts; the lower stays.** The addresses are in every signal and in both nodes' seed lists, so both nodes compute the same answer without talking. Member age (up-number) was rejected: by the time a signal fires both sides have removed each other, so the peer's up-number exists only as remembered history that can differ (an unconverged join, a restart since), and a disagreement means both restart or neither. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. The lower node gives its peer one more `ShutOutGrace` and restarts only if the peer has neither rejoined nor gone quiet by then (the single-sided case where the stale side happened to be the lower address).
|
||||
|
||||
After both sides have removed each other neither sends the other anything, so a healed partition would stay silent: the actor **probes** (an `Identify` to `/system/cluster/core/daemon`) every peer it saw removed after a Down, backing off 5 s → 60 s, until that address is a member again. Each probe of a dead peer logs one Akka association warning.
|
||||
|
||||
"Restart" is the existing self-down path (below): `Cluster.Down(self)` ⇒ the `ActorSystem` terminates ⇒ the process exits **75** ⇒ the supervisor restarts it and it rejoins as standby. Every decision logs a `[leader-auto-down]` line: `shut out by <peer> … tiebreak not needed — this node RESTARTS`, `split from <peer> … Giving it <grace> …`, `Tiebreak says RESTART|STAY`, `rejoined … stays`. Settings: `RestartWhenShutOut` (bool, default `true`) and `ShutOutGrace` (default `60s`, must be positive, must cover the peer's worst-case shutdown). Off restores the pre-#132 behaviour (dual-active until an operator restarts one side). Proven by `ShutOutRestartTests` (unit) and `ShutOutRestartScenarioTests` (real two-node clusters, freeze shape and healed partition); rig evidence in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7.
|
||||
|
||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||
|
||||
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
|
||||
@@ -116,8 +134,8 @@ Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `
|
||||
When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
|
||||
|
||||
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
|
||||
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
|
||||
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
|
||||
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits — with code 75** (`AkkaHostedService.SelfDownExitCode`, `EX_TEMPFAIL`; ScadaBridge#132). Under the Windows service lifetime the code is also set on the service (`ServiceBase.ExitCode`) before the stop, so the SCM sees it.
|
||||
3. The service supervisor restarts it — docker `restart: unless-stopped` (any exit code), or Windows service recovery actions (`sc.exe failure … restart/…` **plus `sc.exe failureflag <svc> 1`**, both set by the wonder install script `deploy/wonder-app-vd03/install.ps1`, which is kept outside git). With `failureflag 1` the SCM runs the recovery actions for a service that stops with a **non-zero** exit code; before #132 the watchdog exit reported 0, which the SCM treats as a clean stop. (ScadaBridge runs as native .NET Windows services; NSSM wraps only the LDAP directory.)
|
||||
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
|
||||
|
||||
### Seed Node Ordering
|
||||
|
||||
Reference in New Issue
Block a user