From 952db02aabeb2c10cded83a8edcc29c07e2acca8 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 12 Nov 2026 20:57:42 -0500 Subject: [PATCH] fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132) A node frozen past failure detection + StableAfter, or both sides of a partition that heals after each downed the other, used to stay a second active node until an operator restarted one: the peer quarantines the removed UID and Akka.Cluster 1.5.62 never acts on ThisActorSystemQuarantinedEvent. ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for leader-auto-down only) watches two signals: - S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer is still a live member here, only this side can be shut out -> restart now. - S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still runs the incarnation this node removed. A healed partition yields only S2, on both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then tiebreak. Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower stays and restarts only if still split one grace later. Removed peers are probed (Identify, 5s -> 60s back-off) since nothing else contacts them after mutual removal. Restart = Cluster.Down(self) -> existing self-down path. The Host now exits 75 after an unplanned ActorSystem termination (and sets it on the Windows service before it reports stopped): SCM recovery with failureflag 1 only restarts a service that stops with a non-zero code, and the watchdog used to exit 0. Shipped default strategy stays auto-down. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi --- docs/components/ClusterInfrastructure.md | 4 +- ...2026-11-09-autodowning-replacement-plan.md | 33 + .../Component-ClusterInfrastructure.md | 22 +- .../ClusterOptions.cs | 55 ++ .../ClusterOptionsValidator.cs | 11 + .../Actors/AkkaHostedService.cs | 59 +- .../Actors/LeaderAutoDowningProvider.cs | 33 +- .../Actors/ShutOutRestart.cs | 674 ++++++++++++++++++ src/ZB.MOM.WW.ScadaBridge.Host/Program.cs | 28 + .../ClusterOptionsValidatorTests.cs | 28 + .../HoconBuilderTests.cs | 42 ++ .../ShutOutRestartTests.cs | 447 ++++++++++++ .../UnexpectedTerminationTests.cs | 16 + .../Cluster/ShutOutRestartScenarioTests.cs | 204 ++++++ 14 files changed, 1646 insertions(+), 10 deletions(-) create mode 100644 src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ShutOutRestartTests.cs create mode 100644 tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ShutOutRestartScenarioTests.cs diff --git a/docs/components/ClusterInfrastructure.md b/docs/components/ClusterInfrastructure.md index 3d1f04d1..90047e16 100644 --- a/docs/components/ClusterInfrastructure.md +++ b/docs/components/ClusterInfrastructure.md @@ -111,7 +111,9 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie ### Downed-node recovery -`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`). +`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits **with code 75** (`AkkaHostedService.SelfDownExitCode`; set by `Program` after the host stops, and on the Windows service itself before it reports `SERVICE_STOPPED`) and the service supervisor (docker `restart: unless-stopped`, Windows service recovery with `failureflag 1`, which only acts on a non-zero exit code) restarts it as a fresh incarnation (`AkkaHostedService.GetOrCreateActorSystem`). + +Under `leader-auto-down` the same path restarts a node that its peer has **shut out** — downed, removed and quarantined it (a frozen node coming back), or both sides of a healed partition, where the higher address restarts. See `Component-ClusterInfrastructure.md` → *Shut-out restart* (ScadaBridge#132). **Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected. diff --git a/docs/plans/2026-11-09-autodowning-replacement-plan.md b/docs/plans/2026-11-09-autodowning-replacement-plan.md index c93ed01a..3966b1b3 100644 --- a/docs/plans/2026-11-09-autodowning-replacement-plan.md +++ b/docs/plans/2026-11-09-autodowning-replacement-plan.md @@ -104,3 +104,36 @@ Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the prove - **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged). - **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs. + +## 7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132) + +The #116 rig drill found a gap that `AutoDowning` and its port share: a node frozen past failure detection plus the window +comes back as a **permanent second active node**, and the same happens to both sides of a partition that heals after each +side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote +publishes `ThisActorSystemQuarantinedEvent` on the stale node, and Akka.Cluster never subscribes to it. + +**Owner decision (#132):** under `leader-auto-down`, a node that is shut out restarts itself; when both sides are shut out, +exactly one restarts, by a deterministic tiebreak. Implemented as `ShutOutRestart` (Host, a child of the downing actor, so +`LeaderAutoDownBase` stays a line-for-line port), behind `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default +`true` for this strategy only) and `ShutOutGrace` (default 60 s). The shipped default strategy stays `auto-down`. + +- **Two signals, confirmed in-process.** S1 `ThisActorSystemQuarantinedEvent` (the frozen node). S2 an `AssociationErrorEvent` + whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed + partition produces **only S2, on both sides** — each side aborts the other's handshake with its own refused UID before a + quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with + `Identify` (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it. +- **Decision.** S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this + node removed the peer: tiebreak. S2: wait `ShutOutGrace` (a frozen peer restarts itself meanwhile), then tiebreak if still + split. Tiebreak: **higher address restarts**, the lower stays and restarts only if the peer is still split from it one + more grace later. +- **Why address, not age:** see `Component-ClusterInfrastructure.md` → *Shut-out restart*. In short, both nodes hold both + addresses always; the peer's up-number is history that can differ after the split. +- **Exit code.** The restart is the existing self-down path, and it exposed a latent gap: the watchdog's `StopApplication()` + exited 0, and Windows service recovery with `failureflag 1` acts only on a non-zero exit. The Host now exits 75 after an + unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code. +- **Contract change for #116 gate 4:** the partition still produces dual-active *while it lasts*; after it heals, one side + (the higher address) restarts and the pair converges with no operator action. + `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds (it never heals). +- **Tests:** `ShutOutRestartTests` (unit), `ShutOutRestartScenarioTests` (two real nodes: freeze shape both address orders, + healed partition, switch off), `UnexpectedTerminationTests`, `HoconBuilderTests`, `ClusterOptionsValidatorTests`. + Rig evidence: `2026-11-12-leader-auto-down-rig-drill.md` §7. diff --git a/docs/requirements/Component-ClusterInfrastructure.md b/docs/requirements/Component-ClusterInfrastructure.md index 207fdf8c..c6540fbb 100644 --- a/docs/requirements/Component-ClusterInfrastructure.md +++ b/docs/requirements/Component-ClusterInfrastructure.md @@ -104,6 +104,24 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`. +#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132) + +Without it, a node that was **frozen** (VM pause, long GC stall, `docker pause`) for longer than failure detection plus `StableAfter` comes back as a **second, permanent active node**: its peer has already downed and removed it, quarantining its UID, and Akka.Cluster 1.5.62 never tells the frozen node — so it downs the peer in its own one-node view. Both sides of a **real partition that heals** after each side downed the other end the same way. This happens under `auto-down` too (Akka core behaviour); the in-repo provider is where it is fixed. + +With `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` on (**the default whenever `leader-auto-down` is selected**), the `ShutOutRestart` actor (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs`, a child of the downing actor) restarts the stale side: + +| Signal (Akka.NET 1.5.62) | Who sees it | Decision | +|---|---|---| +| **S1** `ThisActorSystemQuarantinedEvent` — the peer has quarantined *this* node — while the peer is still a live member in this node's view | the frozen node | **restart now**; no tiebreak (this node has not removed the peer, so the peer cannot be shut out by it) | +| **S1** after this node has removed the peer too | either side of a split | **tiebreak now** | +| **S2** `AssociationErrorEvent` "remote system has a UID that has been quarantined" — the peer still runs the incarnation *this* node removed | both sides of a healed partition; the survivor of a freeze | wait `ShutOutGrace` (a frozen peer restarts itself on S1 meanwhile); still split at the end ⇒ **tiebreak** | + +**Tiebreak: the higher address (host, then port — Akka's `Member.AddressOrdering`) restarts; the lower stays.** The addresses are in every signal and in both nodes' seed lists, so both nodes compute the same answer without talking. Member age (up-number) was rejected: by the time a signal fires both sides have removed each other, so the peer's up-number exists only as remembered history that can differ (an unconverged join, a restart since), and a disagreement means both restart or neither. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. The lower node gives its peer one more `ShutOutGrace` and restarts only if the peer has neither rejoined nor gone quiet by then (the single-sided case where the stale side happened to be the lower address). + +After both sides have removed each other neither sends the other anything, so a healed partition would stay silent: the actor **probes** (an `Identify` to `/system/cluster/core/daemon`) every peer it saw removed after a Down, backing off 5 s → 60 s, until that address is a member again. Each probe of a dead peer logs one Akka association warning. + +"Restart" is the existing self-down path (below): `Cluster.Down(self)` ⇒ the `ActorSystem` terminates ⇒ the process exits **75** ⇒ the supervisor restarts it and it rejoins as standby. Every decision logs a `[leader-auto-down]` line: `shut out by … tiebreak not needed — this node RESTARTS`, `split from … Giving it …`, `Tiebreak says RESTART|STAY`, `rejoined … stays`. Settings: `RestartWhenShutOut` (bool, default `true`) and `ShutOutGrace` (default `60s`, must be positive, must cover the peer's worst-case shutdown). Off restores the pre-#132 behaviour (dual-active until an operator restarts one side). Proven by `ShutOutRestartTests` (unit) and `ShutOutRestartScenarioTests` (real two-node clusters, freeze shape and healed partition); rig evidence in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7. + ### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21) - **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability. @@ -116,8 +134,8 @@ Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. ` When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is: 1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination. -2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**. -3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`). +2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits — with code 75** (`AkkaHostedService.SelfDownExitCode`, `EX_TEMPFAIL`; ScadaBridge#132). Under the Windows service lifetime the code is also set on the service (`ServiceBase.ExitCode`) before the stop, so the SCM sees it. +3. The service supervisor restarts it — docker `restart: unless-stopped` (any exit code), or Windows service recovery actions (`sc.exe failure … restart/…` **plus `sc.exe failureflag 1`**, both set by the wonder install script `deploy/wonder-app-vd03/install.ps1`, which is kept outside git). With `failureflag 1` the SCM runs the recovery actions for a service that stops with a **non-zero** exit code; before #132 the watchdog exit reported 0, which the SCM treats as a clean stop. (ScadaBridge runs as native .NET Windows services; NSSM wraps only the LDAP directory.) 4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below. ### Seed Node Ordering diff --git a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs index c8c6ee77..1bc069b5 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs @@ -144,6 +144,61 @@ public class ClusterOptions /// docs/plans/2026-11-09-tls-production-profile-design.md. /// public ClusterRemotingTlsOptions RemotingTls { get; set; } = new(); + + /// + /// Settings that apply only when is + /// leader-auto-down (ScadaBridge:Cluster:LeaderAutoDown, ScadaBridge#132). + /// Inert under every other strategy: AkkaHostedService.BuildHocon emits none of it. + /// + public LeaderAutoDownOptions LeaderAutoDown { get; set; } = new(); +} + +/// +/// leader-auto-down-only settings (ScadaBridge#132): what a node does when it learns that +/// its pair peer has shut it out — removed and quarantined it, so it can never talk to that +/// peer again until it restarts. +/// +/// +/// +/// Why it exists. A node that was frozen (VM pause, GC stall, docker pause) past the +/// failure-detection window plus is downed and removed by +/// its peer. When it resumes, every association is refused with "the remote system has quarantined +/// this system" — Akka.Remote publishes ThisActorSystemQuarantinedEvent, but Akka.Cluster +/// 1.5.62 never acts on it, so the frozen node downs the peer in its own one-node view and becomes +/// a second, permanent active node. The same thing happens to both nodes after a real +/// network partition heals. +/// +/// +/// What it does (with on): a node that is shut out by its +/// peer while it still counts that peer as a member restarts itself (it is the stale side). When it +/// has already removed the peer too — both sides shut each other out — a deterministic tiebreak +/// both nodes compute identically picks ONE side: the node with the higher address +/// (host, then port; Akka's own member address ordering) restarts, the lower stays. The lower node +/// restarts too only if it is still being shut out after . "Restart" is +/// the existing self-down path: Cluster.Down(self), the ActorSystem terminates, the +/// Host exits non-zero and the service supervisor restarts it as a fresh incarnation that rejoins +/// as standby. Implemented by ShutOutRestart in the Host. +/// +/// +public sealed class LeaderAutoDownOptions +{ + /// + /// Whether a node shut out by its pair peer restarts itself (see the class remarks). Default + /// — on whenever leader-auto-down is selected. Set + /// to get the old behaviour back: a frozen node or a healed partition + /// leaves two active nodes until an operator restarts one. + /// + public bool RestartWhenShutOut { get; set; } = true; + + /// + /// How long the tiebreak WINNER (the lower-address node) waits for its peer to restart before + /// it re-checks. At the end of the grace it restarts only if the peer is still not a member AND + /// it was shut out again in the last few seconds — i.e. the peer never restarted, so this node + /// is the stale side after all. Must cover the peer's worst-case shut-down time (its old + /// incarnation keeps refusing until its ActorSystem has terminated). Default 60s. + /// Validated > 0 at boot when the feature is on. + /// + public TimeSpan ShutOutGrace { get; set; } = TimeSpan.FromSeconds(60); } /// diff --git a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs index 5a653561..8aa1990d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs @@ -74,6 +74,17 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase TimeSpan.Zero, + "ClusterOptions.LeaderAutoDown.ShutOutGrace must be a positive duration when " + + "LeaderAutoDown.RestartWhenShutOut is on (ScadaBridge#132)."); + // DownIfAlone is a keep-oldest knob; under (leader-)auto-down each side downs the // unreachable peer regardless, so the flag is inert and any value is fine. var isKeepOldest = string.Equals( diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs index c65a5f48..a8fe7b8a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs @@ -34,8 +34,20 @@ public class AkkaHostedService : IHostedService private readonly CommunicationOptions _communicationOptions; private readonly ILogger _logger; private readonly IHostApplicationLifetime? _appLifetime; + private readonly IHostLifetime? _hostLifetime; private ActorSystem? _actorSystem; + /// + /// Process exit code after an ActorSystem termination outside host shutdown (a self-down, + /// or a ScadaBridge#132 shut-out restart): 75, EX_TEMPFAIL — "temporary failure, try + /// again". It must be NON-zero: docker restart: unless-stopped restarts on any exit, but + /// Windows service recovery (sc.exe failure … restart with failureflag 1, set by + /// deploy/wonder-app-vd03/install.ps1) applies its restart actions to a service that + /// reports SERVICE_STOPPED only when the reported exit code is not 0. A clean + /// StopApplication() alone reports 0, so the node would stay stopped. + /// + public const int SelfDownExitCode = 75; + /// /// Flips to true the moment begins, so the /// watchdog can tell an @@ -84,14 +96,17 @@ public class AkkaHostedService : IHostedService /// The communication configuration options. /// The logger instance. /// The host application lifetime; optional so existing construction sites (tests, DI variants) keep working without it. + /// The host lifetime; when it is the Windows service lifetime, the self-down exit code is reported to the SCM through it. Optional. public AkkaHostedService( IServiceProvider serviceProvider, IOptions nodeOptions, IOptions clusterOptions, IOptions communicationOptions, ILogger logger, - IHostApplicationLifetime? appLifetime = null) + IHostApplicationLifetime? appLifetime = null, + IHostLifetime? hostLifetime = null) { + _hostLifetime = hostLifetime; _serviceProvider = serviceProvider; _nodeOptions = nodeOptions.Value; _clusterOptions = clusterOptions.Value; @@ -212,8 +227,11 @@ public class AkkaHostedService : IHostedService { if (_stopRequested) return; _logger.LogCritical( - "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). " - + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation."); + "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / " + + "leader-auto-down shut-out restart). Stopping the host process with exit code {ExitCode} so the " + + "service supervisor restarts this node as a fresh incarnation.", SelfDownExitCode); + _terminatedOutsideHostShutdown = true; + MarkServiceSelfDownExit(_hostLifetime); _appLifetime?.StopApplication(); }, TaskContinuationOptions.ExecuteSynchronously); @@ -223,6 +241,37 @@ public class AkkaHostedService : IHostedService } } + /// + /// once the has terminated outside + /// (a self-down or a shut-out restart). Program reads it after + /// RunAsync returns and exits with ; it is a flag rather than a + /// direct write of the process exit code so test hosts that terminate a system never inherit + /// a non-zero process exit code. + /// + public bool TerminatedOutsideHostShutdown => _terminatedOutsideHostShutdown; + + private volatile bool _terminatedOutsideHostShutdown; + + /// + /// When the host runs as a Windows service, sets as the exit code + /// the service reports to the SCM when it stops — the value Windows service recovery + /// (failureflag 1) keys its restart on. Must run BEFORE StopApplication(): the + /// Windows service lifetime reports SERVICE_STOPPED during host shutdown, before + /// Program can set the process exit code. + /// + /// The host lifetime; the Windows service lifetime is a ServiceBase. + /// when the exit code was set on a Windows service lifetime. + internal static bool MarkServiceSelfDownExit(IHostLifetime? hostLifetime) + { + if (OperatingSystem.IsWindows() && hostLifetime is System.ServiceProcess.ServiceBase service) + { + service.ExitCode = SelfDownExitCode; + return true; + } + + return false; + } + /// /// Builds the Akka HOCON configuration document. Every interpolated value is /// routed through (string values) so a hostname, @@ -313,8 +362,12 @@ public class AkkaHostedService : IHostedService // The leader-auto-down duration sits outside akka.cluster so Akka never mistakes it // for one of its own (deprecated) auto-down keys. Only emitted for that strategy, so // the auto-down / keep-oldest documents are byte-identical to before. + // ScadaBridge#132: the shut-out restart switch and grace ride in the same block. + var shutOut = clusterOptions.LeaderAutoDown ?? new LeaderAutoDownOptions(); var leaderAutoDownBlock = string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase) ? $"{LeaderAutoDowningProvider.UnreachableAfterPath} = {DurationHocon(clusterOptions.StableAfter)}\n" + + $"{ShutOutRestartSettings.EnabledPath} = {(shutOut.RestartWhenShutOut ? "on" : "off")}\n" + + $"{ShutOutRestartSettings.GracePath} = {DurationHocon(shutOut.ShutOutGrace)}\n" : string.Empty; // Akka.Remote TLS (ScadaBridge#64, opt-in). Emitted inside dot-netty.tcp only when diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs index 5eb0d3ac..dd689c55 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs @@ -88,7 +88,8 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider $"LeaderAutoDowningProvider selected but '{UnreachableAfterPath}' not set"); } - return LeaderAutoDown.Props(after.Value, _cluster); + return LeaderAutoDown.Props( + after.Value, _cluster, ShutOutRestartSettings.FromConfig(_system.Settings.Config)); } } @@ -118,21 +119,29 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider /// internal sealed class LeaderAutoDown : LeaderAutoDownBase { + /// Child name of the actor. + internal const string ShutOutRestartName = "shut-out-restart"; + private readonly Akka.Cluster.Cluster _cluster; + private readonly ShutOutRestartSettings? _shutOut; /// Props for the downing actor. /// How long a member must stay unreachable before it is downed. /// The cluster extension. - public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) => - Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster)); + /// Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling. + /// The props. + public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut = null) => + Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster, shutOut)); /// Creates the actor. /// How long a member must stay unreachable before it is downed. /// The cluster extension. - public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) + /// Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling. + public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut) : base(unreachableAfter) { _cluster = cluster; + _shutOut = shutOut; } /// @@ -145,6 +154,22 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase protected override void PreStart() { _cluster.Subscribe(Self, typeof(ClusterEvent.IClusterDomainEvent)); + + // ScadaBridge#132: the shut-out restart is a separate child so the downing logic above + // stays a line-for-line port of Akka's AutoDownBase. A child, not a sibling, so it lives + // and dies with this provider's actor and exists only under leader-auto-down. + if (_shutOut is { Enabled: true }) + { + Context.ActorOf(ShutOutRestart.Props(_shutOut, _cluster), ShutOutRestartName); + } + else + { + Context.GetLogger().Info( + "[leader-auto-down] shut-out restart is OFF for {0} (ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut = false): " + + "a node its peer has removed and quarantined keeps running as a second active node until it is restarted", + _cluster.SelfAddress); + } + base.PreStart(); } diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs new file mode 100644 index 00000000..91613cde --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs @@ -0,0 +1,674 @@ +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Akka.Event; +using Akka.Remote; + +namespace ZB.MOM.WW.ScadaBridge.Host.Actors; + +/// +/// Settings for , read from the scadabridge.cluster.leader-auto-down +/// HOCON block that emits from +/// ScadaBridge:Cluster:LeaderAutoDown (ScadaBridge#132). +/// +/// Whether a node that is shut out by its peer restarts itself. +/// How long a peer gets to restart itself before this node acts (see ). +/// First interval between probes of a removed peer, and the fixed interval while a grace runs. +/// Cap for the probe back-off while no grace runs. +internal sealed record ShutOutRestartSettings( + bool Enabled, TimeSpan Grace, TimeSpan ProbeInterval, TimeSpan ProbeMaxInterval) +{ + /// HOCON switch: on/off. Missing = on. + public const string EnabledPath = "scadabridge.cluster.leader-auto-down.restart-when-shut-out"; + + /// HOCON duration for . Missing = 60s. + public const string GracePath = "scadabridge.cluster.leader-auto-down.shut-out-grace"; + + /// HOCON duration for . Missing = 5s. Not bound from appsettings (tests only). + public const string ProbeIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-interval"; + + /// HOCON duration for . Missing = 60s. Not bound from appsettings (tests only). + public const string ProbeMaxIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval"; + + /// + /// How recent the last shut-out evidence must be, when a grace ends, for the split to count as + /// still in place: three probe intervals, so a missed probe or two does not flip the verdict. + /// + public TimeSpan RecentWindow => ProbeInterval * 3; + + /// Reads the settings, defaulting every missing key. + /// The actor system's configuration. + /// The settings. + public static ShutOutRestartSettings FromConfig(Config config) + { + var enabled = !config.HasPath(EnabledPath) || config.GetBoolean(EnabledPath, true); + return new ShutOutRestartSettings( + enabled, + Duration(config, GracePath, TimeSpan.FromSeconds(60)), + Duration(config, ProbeIntervalPath, TimeSpan.FromSeconds(5)), + Duration(config, ProbeMaxIntervalPath, TimeSpan.FromSeconds(60))); + } + + private static TimeSpan Duration(Config config, string path, TimeSpan fallback) => + config.HasPath(path) ? config.GetTimeSpan(path, fallback, true) : fallback; +} + +/// +/// ScadaBridge#132: makes a node that is split from its pair peer — the peer has downed and removed +/// it, or each has removed the other — restart itself, so a frozen node or a healed partition does +/// not leave two active nodes until an operator acts. Runs as a child of the +/// downing actor, so it exists only under leader-auto-down, and +/// only when ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut is on (the default for +/// that strategy). +/// +/// The signals (Akka.NET 1.5.62, read from the decompiled assemblies and confirmed by the +/// in-process scenario tests). When a node removes a downed member, its cluster quarantines that +/// member's UID. After that, each association between the two incarnations fails, and which side +/// learns what depends on who still counts whom as a member: +/// +/// S1 — "the peer has quarantined THIS node". Akka.Remote publishes +/// (EndpointManager, on an +/// InvalidAssociation whose disassociate reason is Quarantined). Seen by a node that +/// still talks to a peer which has removed it: the frozen node after docker pause. Akka.Cluster +/// 1.5.62 never subscribes to it — ClusterCoreDaemon handles only the opposite +/// QuarantinedEvent — which is the root cause of #132. +/// S2 — "the peer is still running the incarnation THIS node removed". An outbound +/// handshake that meets a UID this node has quarantined is aborted (ProtocolStateActor, +/// ForbiddenUidReason: "The remote system has a UID that has been quarantined"), surfacing as +/// an . Seen by both sides of a partition that healed after each +/// side downed the other — neither of them sees S1, because each aborts the other's handshake first — +/// and by the survivor of a freeze when it probes the frozen node. +/// +/// +/// +/// The decision. +/// +/// S1 while the peer is still a live member in this node's view (Joining, WeaklyUp, Up +/// or Leaving): this node has not removed the peer, so it has not quarantined it, so the peer +/// cannot be shut out by this node — only this side is. This is the frozen node. Restart +/// now; no tiebreak is needed. +/// S1 after this node has removed the peer too: both sides are shut out. +/// Tiebreak now (below). +/// S2: the peer's old incarnation is alive. It may be a frozen node about to restart +/// itself on S1 — so this node first gives it . If, at +/// the end, the peer has neither rejoined nor gone quiet, both sides are shut out: tiebreak. +/// +/// Tiebreak: the node with the higher address restarts now. The lower one stays +/// and gives the peer one more ; it restarts only if the +/// peer has still not rejoined and is still producing evidence at the end — the single-sided case +/// where the stale side happened to be the lower address. +/// +/// "Quiet" means no S1/S2 from that peer for : the +/// old incarnation is gone (restarting, or dead), so there is nothing left to split from. +/// +/// +/// Why the address tiebreak, not member age. Both nodes must reach the same answer without +/// talking to each other, from state they already hold. The two addresses are in every signal and +/// in both nodes' seed lists, so the comparison is identical on both sides by construction. Age +/// (up-number) is not: by the time a signal fires both sides have removed each other, so the peer's +/// up-number survives only as remembered history, which differs if a join never converged before +/// the split or either side has restarted since. A disagreement there means both restart or neither +/// does — exactly what the tiebreak exists to prevent. The ordering is Akka's own +/// (host, then port), which also picks the Akka leader of the +/// re-formed pair, and matches the bootstrap guard, where the lower address is the preferred founder. +/// +/// +/// Probing. After both sides have removed each other neither sends the other anything, so a +/// healed partition would stay silent and neither signal would ever fire. This actor therefore +/// sends an to every peer it saw removed after being downed, with a back-off +/// from to +/// (fixed at the short interval while a grace +/// runs), until that address is a member again. A probe of a restarted incarnation (new UID) is +/// harmless — the new incarnation is joining anyway; a probe of the old incarnation produces S2 +/// here and S1 or S2 there. +/// +/// +/// The restart reuses the existing self-down path: Cluster.Down(self) ⇒ +/// run-coordinated-shutdown-when-down ⇒ the ActorSystem terminates ⇒ the Host's +/// WhenTerminated watchdog exits the process with +/// ⇒ the service supervisor (docker restart: unless-stopped, Windows service recovery with +/// failureflag 1) starts a fresh incarnation that rejoins its peer as standby. If the system +/// has not terminated after the Down, CoordinatedShutdown +/// is run directly. +/// +/// +internal sealed class ShutOutRestart : ShutOutRestartBase +{ + /// How long after Cluster.Down(self) the direct CoordinatedShutdown fallback runs. + internal static readonly TimeSpan FallbackShutdownAfter = TimeSpan.FromSeconds(20); + + private readonly Akka.Cluster.Cluster _cluster; + private ICancelable? _fallback; + + /// Props for the actor. + /// The settings. + /// The cluster extension. + /// The props. + public static Props Props(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster) => + Akka.Actor.Props.Create(() => new ShutOutRestart(settings, cluster)); + + /// Creates the actor. + /// The settings. + /// The cluster extension. + public ShutOutRestart(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster) + : base(settings) + { + _cluster = cluster; + } + + /// + public override Address SelfAddress => _cluster.SelfAddress; + + /// + public override IScheduler Scheduler => Context.System.Scheduler; + + /// + protected override IEnumerable
SeedNodes => _cluster.Settings.SeedNodes; + + /// + protected override void PreStart() + { + _cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot, + typeof(ClusterEvent.IMemberEvent)); + Context.System.EventStream.Subscribe(Self, typeof(ThisActorSystemQuarantinedEvent)); + Context.System.EventStream.Subscribe(Self, typeof(AssociationErrorEvent)); + base.PreStart(); + } + + /// + protected override void PostStop() + { + _cluster.Unsubscribe(Self); + Context.System.EventStream.Unsubscribe(Self); + _fallback?.Cancel(); + base.PostStop(); + } + + /// + protected override void RestartSelf() + { + _cluster.Down(_cluster.SelfAddress); + _fallback = Scheduler.ScheduleTellOnceCancelable(FallbackShutdownAfter, Self, FallbackShutdown.Instance, Self); + } + + /// + protected override void Probe(Address peer) + { + Context.ActorSelection(new RootActorPath(peer) / "system" / "cluster" / "core" / "daemon") + .Tell(new Identify(ProbeMessageId), Self); + } + + /// + protected override void OnFallbackShutdown() + { + if (!Context.System.WhenTerminated.IsCompleted) + { + Log.Warning( + "[leader-auto-down] {0} was downed for a shut-out restart {1} ago but its actor system is still running; running CoordinatedShutdown directly", + SelfAddress, FallbackShutdownAfter); + CoordinatedShutdown.Get(Context.System).Run(CoordinatedShutdown.ClusterDowningReason.Instance); + } + } +} + +/// +/// The cluster-free decision logic of , split out (as +/// is) so the tiebreak and the single- and both-sides cases can be +/// unit tested by feeding events straight to the actor. Every decision is logged with the +/// [leader-auto-down] prefix; the restart / stay verdicts at Warning. +/// +internal abstract class ShutOutRestartBase : UntypedActor +{ + /// The carried by every probe. + internal const string ProbeMessageId = "scadabridge-shut-out-probe"; + + /// Timer: probe again. + /// The removed peer to probe. + internal sealed record ProbeTick(Address Peer); + + /// Timer: the grace for has ended. + /// The peer this node is split from. + internal sealed record GraceExpired(Address Peer); + + /// Timer: the direct-shutdown fallback after a restart was requested. + internal sealed class FallbackShutdown + { + /// The singleton instance. + public static readonly FallbackShutdown Instance = new(); + + private FallbackShutdown() + { + } + } + + /// A peer identity independent of the transport scheme (the throttler adapter rewrites it). + private readonly record struct PeerKey(string System, string? Host, int? Port) + { + public static PeerKey Of(Address address) => new(address.System, address.Host, address.Port); + } + + private sealed class ProbeState + { + public required Address Address { get; init; } + public TimeSpan NextInterval { get; set; } + public ICancelable? Timer { get; set; } + } + + /// Phase of a detected split with one peer. + private enum SplitPhase + { + /// S2 seen: the peer's old incarnation is alive; it gets a grace to restart itself. + Suspect, + + /// Tiebreak said stay: the peer is expected to restart; it gets a grace to do so. + Winner, + } + + private sealed class SplitState + { + public required SplitPhase Phase { get; set; } + public required ICancelable Timer { get; set; } + public DateTime LastEvidenceUtc { get; set; } + } + + private readonly ShutOutRestartSettings _settings; + private readonly Dictionary _members = new(); + private readonly HashSet _knownPeers = new(); + private readonly HashSet _ignoredLogged = new(); + private readonly Dictionary _probing = new(); + private readonly Dictionary _splits = new(); + private bool _restarting; + + /// Creates the base logic. + /// The settings. + protected ShutOutRestartBase(ShutOutRestartSettings settings) + { + _settings = settings; + } + + /// Decision log. + protected ILoggingAdapter Log { get; } = Context.GetLogger(); + + /// This node's address. + public abstract Address SelfAddress { get; } + + /// The scheduler used for the probe and grace timers. + public abstract IScheduler Scheduler { get; } + + /// The configured seed nodes; peers of this pair even before they were ever seen as members. + protected abstract IEnumerable
SeedNodes { get; } + + /// The clock; overridable for tests. + protected virtual DateTime UtcNow => DateTime.UtcNow; + + /// Restarts this node (called at most once). + protected abstract void RestartSelf(); + + /// Sends one probe to . + /// The removed peer. + protected abstract void Probe(Address peer); + + /// Runs the direct-shutdown fallback, if the restart has not taken effect. + protected virtual void OnFallbackShutdown() + { + } + + /// + /// Whether an association failure is S2: the handshake met a UID this node has quarantined. + /// Akka.Remote 1.5.62 carries no typed reason outward — the protocol layer's + /// ForbiddenUidReason becomes "The remote system has a UID that has been quarantined. + /// Association aborted." and the endpoint layer's HopelessAssociation says "The remote + /// system [..] has a UID [..] that has been quarantined." — so the cause chain is matched on that + /// wording. Pinned by the in-process scenario tests, which fail if an Akka upgrade rewords it. + /// + /// The association error's cause. + /// when the failure is a refused, quarantined UID. + internal static bool IsQuarantinedUidRefusal(Exception? cause) + { + for (var depth = 0; cause is not null && depth < 8; depth++) + { + var message = cause.Message; + if (message.Contains("has a UID", StringComparison.Ordinal) + && message.Contains("that has been quarantined", StringComparison.Ordinal)) + { + return true; + } + + if (cause is AggregateException aggregate && aggregate.InnerExceptions.Any(IsQuarantinedUidRefusal)) + { + return true; + } + + cause = cause.InnerException; + } + + return false; + } + + /// + protected override void PreStart() + { + foreach (var seed in SeedNodes) + { + AddKnownPeer(seed); + } + + Log.Info( + "[leader-auto-down] shut-out restart is ON for {0}: a node split from its pair peer restarts itself — at once if only it " + + "was removed, otherwise the higher address restarts after a {1} grace (ScadaBridge#132)", + SelfAddress, _settings.Grace); + } + + /// + protected override void PostStop() + { + CancelAllTimers(); + } + + /// + protected override void OnReceive(object message) + { + switch (message) + { + case ClusterEvent.CurrentClusterState state: + _members.Clear(); + foreach (var member in state.Members) + { + TrackMember(member); + } + break; + + case ClusterEvent.MemberRemoved removed: + _members.Remove(PeerKey.Of(removed.Member.Address)); + if (removed.PreviousStatus == MemberStatus.Down && !IsSelf(removed.Member.Address)) + { + StartProbing(removed.Member.Address); + } + break; + + case ClusterEvent.IMemberEvent memberEvent: + TrackMember(memberEvent.Member); + break; + + case ThisActorSystemQuarantinedEvent shutOut: + OnShutOutByPeer(shutOut.RemoteAddress); + break; + + case AssociationErrorEvent error: + if (IsQuarantinedUidRefusal(error.Cause)) + { + OnPeerStillRunningRemovedIncarnation(error.RemoteAddress); + } + break; + + case ProbeTick tick: + OnProbeTick(tick.Peer); + break; + + case GraceExpired expired: + OnGraceExpired(expired.Peer); + break; + + case FallbackShutdown: + OnFallbackShutdown(); + break; + + case ActorIdentity: + // A probe answered by a reachable incarnation — nothing to do. + break; + + default: + Unhandled(message); + break; + } + } + + private static bool IsLiveStatus(MemberStatus status) => + status is MemberStatus.Joining or MemberStatus.WeaklyUp or MemberStatus.Up or MemberStatus.Leaving; + + private bool IsSelf(Address address) => PeerKey.Of(address) == PeerKey.Of(SelfAddress); + + private void AddKnownPeer(Address address) + { + if (!IsSelf(address)) + { + _knownPeers.Add(PeerKey.Of(address)); + } + } + + private void TrackMember(Member member) + { + var key = PeerKey.Of(member.Address); + _members[key] = member; + AddKnownPeer(member.Address); + if (IsLiveStatus(member.Status) && _probing.Remove(key, out var probe)) + { + probe.Timer?.Cancel(); + } + } + + private bool IsLiveMember(Address address, out MemberStatus status) + { + if (_members.TryGetValue(PeerKey.Of(address), out var member) && IsLiveStatus(member.Status)) + { + status = member.Status; + return true; + } + + status = MemberStatus.Removed; + return false; + } + + /// Common gate for both signals: ignore after a restart, for self, and for strangers. + private bool IsRelevantPeer(Address peer, string what) + { + if (_restarting || IsSelf(peer)) + { + return false; + } + + var key = PeerKey.Of(peer); + if (_knownPeers.Contains(key)) + { + return true; + } + + if (_ignoredLogged.Add(key)) + { + Log.Info("[leader-auto-down] {0} {1}, but it is not a seed or member of this cluster; ignored", peer, what); + } + + return false; + } + + /// S1: has quarantined this node. + private void OnShutOutByPeer(Address peer) + { + if (!IsRelevantPeer(peer, $"has quarantined this node ({SelfAddress})")) + { + return; + } + + if (IsLiveMember(peer, out var status)) + { + Log.Warning( + "[leader-auto-down] shut out by {0}: it has removed and quarantined this node ({1}) while this node still counts it as a member ({2}), " + + "so only this side is shut out; tiebreak not needed — this node RESTARTS to rejoin as a fresh incarnation", + peer, SelfAddress, status); + Restart(); + return; + } + + if (_splits.TryGetValue(PeerKey.Of(peer), out var split)) + { + split.LastEvidenceUtc = UtcNow; + if (split.Phase == SplitPhase.Winner) + { + return; + } + + // Suspect → the peer has quarantined us too: both sides are shut out, decide now. + split.Timer.Cancel(); + _splits.Remove(PeerKey.Of(peer)); + } + + Tiebreak(peer, $"shut out by {peer}: it has quarantined this node ({SelfAddress}), and this node has removed it too"); + } + + /// S2: is still running the incarnation this node removed. + private void OnPeerStillRunningRemovedIncarnation(Address peer) + { + if (!IsRelevantPeer(peer, "refused a handshake with a UID this node has quarantined")) + { + return; + } + + var key = PeerKey.Of(peer); + if (_splits.TryGetValue(key, out var split)) + { + split.LastEvidenceUtc = UtcNow; + return; + } + + if (IsLiveMember(peer, out _)) + { + // A live member with a quarantined UID is a transient of Akka's own quarantine path + // (it is about to be marked terminated and downed); the downing provider handles it. + return; + } + + Log.Warning( + "[leader-auto-down] split from {0}: it is still running the incarnation this node ({1}) removed, so this node has shut it out. " + + "Giving it {2} to restart itself before the tiebreak (a frozen node that was removed restarts on its own)", + peer, SelfAddress, _settings.Grace); + StartSplit(peer, SplitPhase.Suspect); + } + + private void Tiebreak(Address peer, string why) + { + if (Member.AddressOrdering.Compare(SelfAddress, peer) > 0) + { + Log.Warning( + "[leader-auto-down] {0}; both sides are shut out. Tiebreak says RESTART: this node ({1}) has the higher address", + why, SelfAddress); + Restart(); + return; + } + + Log.Warning( + "[leader-auto-down] {0}; both sides are shut out. Tiebreak says STAY: this node ({1}) has the lower address, so {2} restarts. " + + "This node restarts too only if {2} has not rejoined and is still split from it after {3}", + why, SelfAddress, peer, _settings.Grace); + StartSplit(peer, SplitPhase.Winner); + } + + private void StartSplit(Address peer, SplitPhase phase) + { + _splits[PeerKey.Of(peer)] = new SplitState + { + Phase = phase, + Timer = Scheduler.ScheduleTellOnceCancelable(_settings.Grace, Self, new GraceExpired(peer), Self), + LastEvidenceUtc = UtcNow, + }; + + // Probe at the short interval during the grace, so "still split" is measured. + StartProbing(peer, restartBackOff: true); + } + + private void OnGraceExpired(Address peer) + { + if (_restarting || !_splits.Remove(PeerKey.Of(peer), out var split)) + { + return; + } + + if (IsLiveMember(peer, out var status)) + { + Log.Info( + "[leader-auto-down] {0} rejoined ({1}) within the shut-out grace; this node ({2}) stays", + peer, status, SelfAddress); + return; + } + + var sinceLast = UtcNow - split.LastEvidenceUtc; + if (sinceLast > _settings.RecentWindow) + { + Log.Info( + "[leader-auto-down] {0} has not rejoined yet, but nothing of its old incarnation has been seen for {1:0}s (it is restarting or down); " + + "this node ({2}) stays", + peer, sinceLast.TotalSeconds, SelfAddress); + return; + } + + if (split.Phase == SplitPhase.Suspect) + { + Tiebreak(peer, + $"split from {peer}: its old incarnation is still running {_settings.Grace} later and it has not restarted itself"); + return; + } + + Log.Warning( + "[leader-auto-down] {0} is still split from this node ({1}) {2} after the tiebreak (last seen {3:0}s ago) and has not rejoined, " + + "so this node is the stale side after all; this node RESTARTS", + peer, SelfAddress, _settings.Grace, sinceLast.TotalSeconds); + Restart(); + } + + private void StartProbing(Address peer, bool restartBackOff = false) + { + var key = PeerKey.Of(peer); + if (_probing.TryGetValue(key, out var existing)) + { + if (!restartBackOff) + { + return; + } + + existing.Timer?.Cancel(); + } + + var state = existing ?? new ProbeState { Address = peer }; + state.NextInterval = _settings.ProbeInterval; + state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self); + _probing[key] = state; + } + + private void OnProbeTick(Address peer) + { + var key = PeerKey.Of(peer); + if (_restarting || !_probing.TryGetValue(key, out var state)) + { + return; + } + + Probe(state.Address); + + // Fixed short interval while a grace runs (to measure "still split"); otherwise back off. + state.NextInterval = _splits.ContainsKey(key) + ? _settings.ProbeInterval + : TimeSpan.FromTicks(Math.Min(state.NextInterval.Ticks * 2, _settings.ProbeMaxInterval.Ticks)); + state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self); + } + + private void Restart() + { + _restarting = true; + CancelAllTimers(); + _probing.Clear(); + _splits.Clear(); + RestartSelf(); + } + + private void CancelAllTimers() + { + foreach (var probe in _probing.Values) + { + probe.Timer?.Cancel(); + } + + foreach (var split in _splits.Values) + { + split.Timer.Cancel(); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs index b6e85cc1..b6e172ed 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Program.cs @@ -157,6 +157,11 @@ Log.Logger = ZB.MOM.WW.ScadaBridge.Host.ScadaBridgeLogging // The catch below uses it to tell that routine stop from a genuine fatal exit (ScadaBridge#76). var hostStopping = CancellationToken.None; +// The Akka hosted service, captured with hostStopping (RunAsync disposes the host's service +// provider), so the finally block can turn an Akka watchdog stop (self-down / ScadaBridge#132 +// shut-out restart) into a non-zero exit code whichever way RunAsync unwound. +AkkaHostedService? akkaHostedService = null; + try { Log.Information("Starting ScadaBridge host as {Role} on {Hostname}", nodeRole, nodeHostname); @@ -670,6 +675,7 @@ try var app = builder.Build(); hostStopping = app.Lifetime.ApplicationStopping; + akkaHostedService = app.Services.GetService(); // Apply or validate database migrations (skip when running in test harness) if (!string.Equals(configuration["ScadaBridge:Database:SkipMigrations"], "true", StringComparison.OrdinalIgnoreCase)) @@ -864,6 +870,7 @@ try var app = builder.Build(); hostStopping = app.Lifetime.ApplicationStopping; + akkaHostedService = app.Services.GetService(); // Endpoint routing middleware. The gRPC service mapping below and the // /metrics scrape endpoint both run on endpoint routing, so UseRouting() @@ -934,6 +941,8 @@ catch (Exception ex) } finally { + Program.SetSelfDownExitCode(akkaHostedService); + await Log.CloseAndFlushAsync(); } @@ -955,6 +964,25 @@ public partial class Program internal static bool IsGracefulShutdown(Exception ex, CancellationToken hostStopping) => ex is OperationCanceledException && hostStopping.IsCancellationRequested; + /// + /// After RunAsync returns: when the stop was the Akka watchdog's (the ActorSystem + /// terminated outside host shutdown — a self-down or a ScadaBridge#132 shut-out restart), exit + /// with the non-zero so the service supervisor + /// restarts the node. Docker restart: unless-stopped restarts on any exit; Windows service + /// recovery gets the code through the service lifetime (AkkaHostedService.MarkServiceSelfDownExit). + /// + /// The host's Akka hosted service; null if the host never built. + internal static void SetSelfDownExitCode(AkkaHostedService? akka) + { + if (akka is { TerminatedOutsideHostShutdown: true }) + { + Environment.ExitCode = AkkaHostedService.SelfDownExitCode; + Log.Warning( + "ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code {ExitCode} so the service supervisor restarts it", + AkkaHostedService.SelfDownExitCode); + } + } + /// /// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the /// ASPNETCORE_URLS / --urls value). Port-only view of diff --git a/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs index 41abedfb..639b2dcf 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs @@ -327,4 +327,32 @@ public class ClusterOptionsValidatorTests Assert.True(result.Failed); Assert.Contains("'leader-auto-down'", result.FailureMessage); } + + /// + /// ScadaBridge#132: a non-positive shut-out grace is refused under leader-auto-down with the + /// restart on (it would make both sides of a healed partition restart together), and ignored + /// when the restart is off or another strategy is selected. + /// + /// The strategy. + /// The shut-out restart switch. + /// Whether validation must pass. + [Theory] + [InlineData("leader-auto-down", true, false)] + [InlineData("leader-auto-down", false, true)] + [InlineData("auto-down", true, true)] + public void LeaderAutoDown_ZeroShutOutGrace(string strategy, bool restart, bool expectValid) + { + var options = ValidOptions(); + options.SplitBrainResolverStrategy = strategy; + options.LeaderAutoDown = new LeaderAutoDownOptions { RestartWhenShutOut = restart, ShutOutGrace = TimeSpan.Zero }; + + var result = new ClusterOptionsValidator().Validate(null, options); + + Assert.Equal(expectValid, result.Succeeded); + if (!expectValid) + { + Assert.Contains("ShutOutGrace", result.FailureMessage); + } + } } + diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs index a1f93433..dccab5b2 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs @@ -322,4 +322,46 @@ public class HoconBuilderTests Assert.DoesNotContain("leader-auto-down", hocon); Assert.DoesNotContain(nameof(LeaderAutoDowningProvider), hocon); } + + /// + /// ScadaBridge#132: leader-auto-down carries the shut-out restart switch (default ON) and its + /// grace into the provider's HOCON block, and the provider reads back exactly those values. + /// + /// The configured switch. + [Theory] + [InlineData(true)] + [InlineData(false)] + public void BuildHocon_LeaderAutoDown_EmitsShutOutRestartSettings(bool enabled) + { + var cluster = DefaultCluster(); + cluster.SplitBrainResolverStrategy = "leader-auto-down"; + cluster.LeaderAutoDown = new LeaderAutoDownOptions + { + RestartWhenShutOut = enabled, + ShutOutGrace = TimeSpan.FromMilliseconds(45500), + }; + + var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon( + DefaultNode(), cluster, new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15))); + + var settings = ShutOutRestartSettings.FromConfig(config); + Assert.Equal(enabled, settings.Enabled); + Assert.Equal(TimeSpan.FromMilliseconds(45500), settings.Grace); + } + + /// ScadaBridge#132: the shut-out restart is ON by default when leader-auto-down is selected. + [Fact] + public void BuildHocon_LeaderAutoDown_ShutOutRestartOnByDefault() + { + var cluster = DefaultCluster(); + cluster.SplitBrainResolverStrategy = "leader-auto-down"; + + var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon( + DefaultNode(), cluster, new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15))); + + Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath)); + Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath)); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ShutOutRestartTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ShutOutRestartTests.cs new file mode 100644 index 00000000..dcb1f95b --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ShutOutRestartTests.cs @@ -0,0 +1,447 @@ +using System.Collections.Immutable; +using System.Reflection; +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Akka.Remote; +using Akka.TestKit.Xunit2; +using Akka.Util; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// ScadaBridge#132: the decision logic of . Cluster events and +/// are fed straight to the actor; the restart and +/// probe seams report to the test actor. Covers the single-side shut-out (restart, no tiebreak), +/// the tiebreak in both orders, the both-sides case where exactly one of two nodes restarts, the +/// winner's grace, and the probing that makes a healed partition produce the event at all. The +/// real two-node behaviour is proven in IntegrationTests (ShutOutRestartScenarioTests). +/// +public class ShutOutRestartTests : TestKit +{ + /// INFO so the decision-log assertions see every line. + public ShutOutRestartTests() + : base("akka.loglevel = INFO") + { + } + + // "a" < "b" < "c" by Akka's member address ordering (host, then port). + private static readonly Address NodeA = new("akka.tcp", "scadabridge", "a", 8081); + private static readonly Address NodeB = new("akka.tcp", "scadabridge", "b", 8081); + private static readonly Address Stranger = new("akka.tcp", "scadabridge", "z", 9999); + + private static readonly ShutOutRestartSettings Fast = new( + Enabled: true, + Grace: TimeSpan.FromSeconds(1.5), + ProbeInterval: TimeSpan.FromMilliseconds(200), + ProbeMaxInterval: TimeSpan.FromMilliseconds(800)); + + private sealed record Restarted(Address Self); + + private sealed record Probed(Address Self, Address Peer); + + /// Test double: restart and probe report to a probe actor. + private sealed class TestShutOutRestart : ShutOutRestartBase + { + private readonly IActorRef _probe; + private readonly Address[] _seeds; + + public TestShutOutRestart(Address self, Address[] seeds, ShutOutRestartSettings settings, IActorRef probe) + : base(settings) + { + SelfAddress = self; + _seeds = seeds; + _probe = probe; + } + + public override Address SelfAddress { get; } + + public override IScheduler Scheduler => Context.System.Scheduler; + + protected override IEnumerable
SeedNodes => _seeds; + + protected override void RestartSelf() => _probe.Tell(new Restarted(SelfAddress)); + + protected override void Probe(Address peer) => _probe.Tell(new Probed(SelfAddress, peer)); + } + + private static Member CreateMember(Address address, MemberStatus status) + { + var create = typeof(Member).GetMethod( + "Create", + BindingFlags.Static | BindingFlags.NonPublic, + new[] { typeof(UniqueAddress), typeof(int), typeof(MemberStatus), typeof(ImmutableHashSet), typeof(AppVersion) }) + ?? throw new InvalidOperationException("Akka.Cluster.Member.Create(…) not found — Akka internals changed."); + return (Member)create.Invoke(null, new object[] + { + new UniqueAddress(address, 1), 1, status, ImmutableHashSet.Empty, AppVersion.Zero, + })!; + } + + private static ClusterEvent.CurrentClusterState StateWith(params Member[] members) => + new( + ImmutableSortedSet.CreateRange(members), + ImmutableHashSet.Empty, + ImmutableHashSet
.Empty, + null, + ImmutableDictionary.Empty); + + private IActorRef Node(Address self, Address peer, IActorRef? probe = null, ShutOutRestartSettings? settings = null) => + Sys.ActorOf(Props.Create(() => new TestShutOutRestart( + self, new[] { self, peer }, settings ?? Fast, probe ?? TestActor))); + + /// Both nodes see each other Up, then each removes the other after downing it (the partition outcome). + private static void BothRemovedEachOther(IActorRef node, Address self, Address peer) + { + node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up))); + node.Tell(new ClusterEvent.MemberRemoved(CreateMember(peer, MemberStatus.Removed), MemberStatus.Down)); + } + + private static ThisActorSystemQuarantinedEvent ShutOutBy(Address peer, Address self) => new(self, peer); + + /// S2 as Akka.Remote 1.5.62 surfaces it: the endpoint writer's InvalidAssociation wrapping the protocol refusal. + private static AssociationErrorEvent StillRunningRemoved(Address peer, Address self) => + new( + new Exception( + $"Association failed with {peer}", + new AggregateException(new Exception("The remote system has a UID that has been quarantined. Association aborted."))), + self, peer, inbound: false, Akka.Event.LogLevel.WarningLevel); + + /// Everything receives during . + private static List Drain(Akka.TestKit.TestProbe probe, TimeSpan window) + { + var received = new List(); + var until = DateTime.UtcNow + window; + while (true) + { + var left = until - DateTime.UtcNow; + if (left <= TimeSpan.Zero) + { + return received; + } + + if (probe.TryReceiveOne(out var envelope, left)) + { + received.Add(envelope.Message); + } + } + } + + /// + /// Single side (the frozen node): the peer is still a live member in this node's view, so only + /// this side can be shut out — it restarts at once, whichever address is higher. + /// + /// Whether this node is the lower-address node. + [Theory] + [InlineData(true)] + [InlineData(false)] + public void ShutOut_PeerStillAMember_RestartsWithoutTiebreak(bool selfIsA) + { + var (self, peer) = selfIsA ? (NodeA, NodeB) : (NodeB, NodeA); + var node = Node(self, peer); + node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up))); + + EventFilter.Warning(contains: "tiebreak not needed").ExpectOne(() => node.Tell(ShutOutBy(peer, self))); + ExpectMsg(new Restarted(self)); + ExpectNoMsg(TimeSpan.FromMilliseconds(300)); + } + + /// Tiebreak, higher address: both sides removed each other; the higher address restarts at once. + [Fact] + public void ShutOut_PeerRemoved_HigherAddress_Restarts() + { + var node = Node(NodeB, NodeA); + BothRemovedEachOther(node, NodeB, NodeA); + + EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB))); + ExpectMsg(new Restarted(NodeB)); + } + + /// + /// Tiebreak, lower address: it stays; it keeps probing during the grace, and at the end of the + /// grace — the peer silent, i.e. it restarted — it still stays. + /// + [Fact] + public void ShutOut_PeerRemoved_LowerAddress_Stays_AndStaysAfterGraceWhenPeerWentQuiet() + { + var probe = CreateTestProbe(); + var node = Node(NodeA, NodeB, probe.Ref); + BothRemovedEachOther(node, NodeA, NodeB); + + EventFilter.Warning(contains: "Tiebreak says STAY").ExpectOne(() => node.Tell(ShutOutBy(NodeB, NodeA))); + EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { }); + + var received = Drain(probe, TimeSpan.FromMilliseconds(500)); + Assert.Contains(new Probed(NodeA, NodeB), received); // it kept probing its peer + Assert.DoesNotContain(new Restarted(NodeA), received); // and never restarted + } + + /// + /// Tiebreak, lower address, but the peer keeps refusing for the whole grace and never rejoins: + /// the lower node was the stale side after all (single-sided with the lower address) and restarts. + /// + [Fact] + public async Task ShutOut_PeerRemoved_LowerAddress_StillRefusedAfterGrace_Restarts() + { + var probe = CreateTestProbe(); + var node = Node(NodeA, NodeB, probe.Ref); + BothRemovedEachOther(node, NodeA, NodeB); + node.Tell(ShutOutBy(NodeB, NodeA)); + + var until = DateTime.UtcNow + Fast.Grace + TimeSpan.FromSeconds(1); + var restarted = false; + while (DateTime.UtcNow < until && !restarted) + { + node.Tell(ShutOutBy(NodeB, NodeA)); // every probe is refused + await Task.Delay(100); + while (probe.HasMessages) + { + restarted |= probe.ReceiveOne(TimeSpan.Zero) is Restarted; + } + } + + Assert.True(restarted, "the lower node never restarted although its peer kept refusing it past the grace"); + } + + /// Tiebreak, lower address: the peer rejoins during the grace; the lower node stays. + [Fact] + public void ShutOut_PeerRemoved_LowerAddress_PeerRejoins_Stays() + { + var probe = CreateTestProbe(); + var node = Node(NodeA, NodeB, probe.Ref); + BothRemovedEachOther(node, NodeA, NodeB); + node.Tell(ShutOutBy(NodeB, NodeA)); + node.Tell(ShutOutBy(NodeB, NodeA)); // refused right up to the restart of the peer + node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeB, MemberStatus.Up))); + + EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { }); + Assert.DoesNotContain(new Restarted(NodeA), Drain(probe, TimeSpan.FromMilliseconds(500))); + } + + /// + /// Both sides shut out (a healed partition): two nodes, each having removed the other, both + /// receive the event — exactly ONE restarts, and it is the higher address. Run in both + /// delivery orders so the outcome cannot depend on who hears first. + /// + /// Whether the higher-address node receives its event first. + [Theory] + [InlineData(true)] + [InlineData(false)] + public void BothSidesShutOut_ExactlyOneRestarts_TheHigherAddress(bool higherHearsFirst) + { + var probe = CreateTestProbe(); + var a = Node(NodeA, NodeB, probe.Ref); + var b = Node(NodeB, NodeA, probe.Ref); + BothRemovedEachOther(a, NodeA, NodeB); + BothRemovedEachOther(b, NodeB, NodeA); + + if (higherHearsFirst) + { + b.Tell(ShutOutBy(NodeA, NodeB)); + a.Tell(ShutOutBy(NodeB, NodeA)); + } + else + { + a.Tell(ShutOutBy(NodeB, NodeA)); + b.Tell(ShutOutBy(NodeA, NodeB)); + } + + // Within the grace, the only restart is B's. + var restarts = Drain(probe, Fast.Grace - TimeSpan.FromMilliseconds(300)).OfType().ToList(); + Assert.Equal(new[] { new Restarted(NodeB) }, restarts); + } + + /// The same event delivered many times restarts once. + [Fact] + public void RepeatedShutOut_RestartsOnce() + { + var node = Node(NodeB, NodeA); + node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up))); + for (var i = 0; i < 5; i++) + { + node.Tell(ShutOutBy(NodeA, NodeB)); + } + + ExpectMsg(new Restarted(NodeB)); + ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + } + + /// A quarantine by an address that is neither a seed nor a member is not this pair's business. + [Fact] + public void ShutOutByStranger_IsIgnored() + { + var node = Node(NodeB, NodeA); + node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up))); + + EventFilter.Info(contains: "not a seed or member of this cluster").ExpectOne(() => node.Tell(ShutOutBy(Stranger, NodeB))); + ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + } + + /// + /// A peer removed after being DOWNED is probed with a back-off until it is a member again; a + /// peer that LEFT (Exiting → Removed) is not probed. + /// + [Fact] + public void DownedPeer_IsProbedUntilItRejoins_LeftPeerIsNot() + { + var node = Node(NodeA, NodeB); + node.Tell(StateWith(CreateMember(NodeA, MemberStatus.Up), CreateMember(NodeB, MemberStatus.Up))); + + node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Exiting)); + ExpectNoMsg(TimeSpan.FromMilliseconds(600)); + + node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Down)); + ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1)); + ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1)); + + node.Tell(new ClusterEvent.MemberJoined(CreateMember(NodeB, MemberStatus.Joining))); + ReceiveWhile(TimeSpan.FromMilliseconds(100), m => m); // a tick already in flight + ExpectNoMsg(TimeSpan.FromSeconds(1.5)); + } + + /// + /// S2 on the higher node (it removed a peer whose old incarnation is still alive): no restart + /// during the grace — a frozen peer gets the chance to restart itself — then, still split, the + /// tiebreak restarts it. + /// + [Fact] + public async Task StillRunningRemoved_HigherAddress_WaitsGrace_ThenRestartsIfStillSplit() + { + var probe = CreateTestProbe(); + var node = Node(NodeB, NodeA, probe.Ref); + BothRemovedEachOther(node, NodeB, NodeA); + + EventFilter.Warning(contains: "Giving it").ExpectOne(() => node.Tell(StillRunningRemoved(NodeA, NodeB))); + var started = DateTime.UtcNow; + var restartedAt = (DateTime?)null; + while (DateTime.UtcNow - started < Fast.Grace + TimeSpan.FromSeconds(1.5) && restartedAt is null) + { + node.Tell(StillRunningRemoved(NodeA, NodeB)); // every probe still meets the old incarnation + await Task.Delay(100); + while (probe.HasMessages) + { + if (probe.ReceiveOne(TimeSpan.Zero) is Restarted) + { + restartedAt = DateTime.UtcNow; + } + } + } + + Assert.NotNull(restartedAt); + Assert.True(restartedAt.Value - started >= Fast.Grace - TimeSpan.FromMilliseconds(200), + $"restarted after {(restartedAt.Value - started).TotalSeconds:0.0}s, before the {Fast.Grace} grace"); + } + + /// + /// The freeze survivor: S2 (the frozen node's old incarnation is alive), then the frozen node + /// restarts itself and its old incarnation goes quiet — the survivor does NOT restart, even + /// though it has the higher address. + /// + [Fact] + public void StillRunningRemoved_PeerGoesQuiet_NoRestartEvenWhenHigher() + { + var probe = CreateTestProbe(); + var node = Node(NodeB, NodeA, probe.Ref); + BothRemovedEachOther(node, NodeB, NodeA); + node.Tell(StillRunningRemoved(NodeA, NodeB)); + + EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { }); + Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500))); + } + + /// S2, then the peer's new incarnation joins during the grace: no restart. + [Fact] + public void StillRunningRemoved_PeerRejoins_NoRestart() + { + var probe = CreateTestProbe(); + var node = Node(NodeB, NodeA, probe.Ref); + BothRemovedEachOther(node, NodeB, NodeA); + node.Tell(StillRunningRemoved(NodeA, NodeB)); + node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeA, MemberStatus.Up))); + node.Tell(StillRunningRemoved(NodeA, NodeB)); // stale evidence from before the restart + + EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { }); + Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500))); + } + + /// + /// A healed partition as Akka 1.5.62 really reports it — S2 on BOTH sides, no S1 — ends with + /// exactly one restart, the higher address, after the grace; the lower stays. + /// + [Fact] + public async Task BothSidesStillRunningRemoved_ExactlyOneRestarts_TheHigherAddress() + { + var probe = CreateTestProbe(); + var a = Node(NodeA, NodeB, probe.Ref); + var b = Node(NodeB, NodeA, probe.Ref); + BothRemovedEachOther(a, NodeA, NodeB); + BothRemovedEachOther(b, NodeB, NodeA); + + var restarts = new List(); + var until = DateTime.UtcNow + Fast.Grace + Fast.Grace - TimeSpan.FromMilliseconds(400); + while (DateTime.UtcNow < until) + { + if (!restarts.Contains(new Restarted(NodeB))) + { + // Until B's old incarnation is gone, each side's probes meet the other's. + a.Tell(StillRunningRemoved(NodeB, NodeA)); + b.Tell(StillRunningRemoved(NodeA, NodeB)); + } + + await Task.Delay(100); + while (probe.HasMessages) + { + if (probe.ReceiveOne(TimeSpan.Zero) is Restarted r) + { + restarts.Add(r); + } + } + } + + Assert.Equal(new[] { new Restarted(NodeB) }, restarts); + } + + /// S1 arriving while an S2 grace runs: the peer has quarantined us too, so the tiebreak runs at once. + [Fact] + public void StillRunningRemoved_ThenShutOut_TiebreaksImmediately() + { + var node = Node(NodeB, NodeA); + BothRemovedEachOther(node, NodeB, NodeA); + node.Tell(StillRunningRemoved(NodeA, NodeB)); + + EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB))); + FishForMessage(m => m is Restarted, TimeSpan.FromSeconds(1)); + } + + /// S2 detection matches Akka 1.5.62's two wordings anywhere in the cause chain, and nothing else. + [Fact] + public void IsQuarantinedUidRefusal_MatchesAkkaWordingOnly() + { + Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal( + new Exception("Association failed with x", new AggregateException( + new Exception("The remote system has a UID that has been quarantined. Association aborted."))))); + Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal( + new Exception("The remote system [akka.tcp://scadabridge@b:8081] has a UID [42] that has been quarantined. Association aborted."))); + Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal( + new Exception("Association failed with x", new Exception("Connection refused")))); + Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal( + new Exception("The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted."))); + Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(null)); + } + + /// Settings: defaults when keys are missing; the switch and durations are read when present. + [Fact] + public void Settings_FromConfig_DefaultsAndOverrides() + { + var defaults = ShutOutRestartSettings.FromConfig(ConfigurationFactory.Empty); + Assert.Equal(new ShutOutRestartSettings(true, TimeSpan.FromSeconds(60), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60)), defaults); + + var set = ShutOutRestartSettings.FromConfig(ConfigurationFactory.ParseString($@" +{ShutOutRestartSettings.EnabledPath} = off +{ShutOutRestartSettings.GracePath} = 2500ms +{ShutOutRestartSettings.ProbeIntervalPath} = 1s +{ShutOutRestartSettings.ProbeMaxIntervalPath} = 4s")); + Assert.Equal(new ShutOutRestartSettings(false, TimeSpan.FromMilliseconds(2500), TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(4)), set); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs index c685c093..ebc311c8 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/UnexpectedTerminationTests.cs @@ -69,6 +69,8 @@ public sealed class UnexpectedTerminationTests await Task.Delay(500); lifetime.Received(1).StopApplication(); + // ScadaBridge#132: Program turns this flag into the non-zero SelfDownExitCode. + Assert.True(service.TerminatedOutsideHostShutdown); } /// Verifies that a normal stop does not trigger an application stop. @@ -84,5 +86,19 @@ public sealed class UnexpectedTerminationTests await Task.Delay(500); lifetime.DidNotReceive().StopApplication(); + Assert.False(service.TerminatedOutsideHostShutdown); + } + + /// + /// ScadaBridge#132: the self-down exit code is non-zero (Windows service recovery with + /// failureflag 1 restarts only on a non-zero exit) and is only pushed into a Windows service + /// lifetime — a console / docker lifetime is left alone. + /// + [Fact] + public void SelfDownExitCode_IsNonZero_AndOnlyAppliedToAServiceLifetime() + { + Assert.NotEqual(0, AkkaHostedService.SelfDownExitCode); + Assert.False(AkkaHostedService.MarkServiceSelfDownExit(null)); + Assert.False(AkkaHostedService.MarkServiceSelfDownExit(Substitute.For())); } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ShutOutRestartScenarioTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ShutOutRestartScenarioTests.cs new file mode 100644 index 00000000..af60a2a5 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/ShutOutRestartScenarioTests.cs @@ -0,0 +1,204 @@ +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Akka.Remote; +using Akka.Remote.Transport; +using Xunit; +using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +/// +/// ScadaBridge#132, on real in-process two-node clusters built from the production +/// BuildHocon (same harness as , with Akka's +/// throttler transport to cut and restore the network without killing either node): +/// +/// the FROZEN-node shape — only one side downed the other before the network came back — +/// ends with the shut-out node restarting (its ActorSystem terminates) and the other +/// staying Up; +/// a partition held past the window and then HEALED — each side downed the other — ends +/// with exactly one restart, the higher address; a fresh incarnation on that address rejoins the +/// survivor as a 2-member cluster, and the survivor is still Up after its grace; +/// with the switch OFF, the healed partition keeps both sides up (the #116 gate-4 contract +/// before #132) — the positive control that the restart comes from the feature. +/// +/// "Restart" in-process is the termination of the node's ActorSystem; the Host turns that +/// into a process exit (covered by UnexpectedTerminationTests). +/// +public sealed class ShutOutRestartScenarioTests : IAsyncLifetime +{ + private static readonly TimeSpan Heartbeat = TimeSpan.FromMilliseconds(500); + private static readonly TimeSpan Threshold = TimeSpan.FromSeconds(2); + + private readonly List _systems = new(); + + private ActorSystem StartNode( + int selfPort, int peerPort, TimeSpan stableAfter, bool restartWhenShutOut = true, TimeSpan? grace = null) + { + const string scheme = "akka.trttl.tcp"; + var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort }; + var clusterOptions = new ClusterOptions + { + SeedNodes = new List + { + $"{scheme}://scadabridge@127.0.0.1:{selfPort}", + $"{scheme}://scadabridge@127.0.0.1:{peerPort}", + }, + SplitBrainResolverStrategy = "leader-auto-down", + StableAfter = stableAfter, + HeartbeatInterval = Heartbeat, + FailureDetectionThreshold = Threshold, + MinNrOfMembers = 1, + LeaderAutoDown = new LeaderAutoDownOptions + { + RestartWhenShutOut = restartWhenShutOut, + ShutOutGrace = grace ?? TimeSpan.FromSeconds(20), + }, + }; + var hocon = AkkaHostedService.BuildHocon( + nodeOptions, clusterOptions, new[] { "Central" }, + TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(60)); + var overrides = "akka.coordinated-shutdown.run-by-actor-system-terminate = off\n" + + "akka.remote.dot-netty.tcp.applied-adapters = [trttl]\n" + + "akka.remote.retry-gate-closed-for = 500ms\n" + // Fast probes so a healed partition is noticed within the test's patience. + + "scadabridge.cluster.leader-auto-down.shut-out-probe-interval = 1s\n" + + "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval = 2s\n"; + var config = ConfigurationFactory.ParseString(overrides) + .WithFallback(ConfigurationFactory.ParseString(hocon)); + var system = ActorSystem.Create("scadabridge", config); + _systems.Add(system); + return system; + } + + private static Task Throttle(ActorSystem from, ActorSystem to, ThrottleMode mode) => + ((RemoteActorRefProvider)((ExtendedActorSystem)from).Provider).Transport.ManagementCommand( + new SetThrottle(Akka.Cluster.Cluster.Get(to).SelfAddress, ThrottleTransportAdapter.Direction.Both, mode)); + + private async Task<(ActorSystem Lower, ActorSystem Higher, int LowerPort, int HigherPort)> StartPair( + TimeSpan stableLower, TimeSpan stableHigher, bool restartWhenShutOut = true) + { + var p1 = TwoNodeClusterFixture.GetFreeTcpPort(); + var p2 = TwoNodeClusterFixture.GetFreeTcpPort(); + var (lowerPort, higherPort) = p1 < p2 ? (p1, p2) : (p2, p1); + var lower = StartNode(lowerPort, higherPort, stableLower, restartWhenShutOut); + await TwoNodeClusterFixture.WaitForMembersUp(lower, 1, TimeSpan.FromSeconds(30)); + var higher = StartNode(higherPort, lowerPort, stableHigher, restartWhenShutOut); + await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMembersUp(higher, 2, TimeSpan.FromSeconds(30)); + return (lower, higher, lowerPort, higherPort); + } + + private static async Task Terminates(ActorSystem system, TimeSpan within) + { + var done = await Task.WhenAny(system.WhenTerminated, Task.Delay(within)); + return done == system.WhenTerminated; + } + + /// + /// Frozen-node shape: only the LOWER node downs the higher one (the higher one's window is far + /// longer than the cut), then the network returns. The higher node — still counting the lower as + /// a member — is refused, restarts without a tiebreak; the lower node stays Up. Then the same with + /// roles swapped, so the single-side rule is proven independent of the address order. + /// + /// Whether the shut-out (frozen) node has the higher address. + /// A task that represents the asynchronous operation. + [Theory] + [InlineData(true)] + [InlineData(false)] + public async Task SingleSideShutOut_ShutOutNodeRestarts_OtherStaysUp(bool shutOutIsHigher) + { + var quick = TimeSpan.FromSeconds(2); + var never = TimeSpan.FromMinutes(5); + var (lower, higher, _, _) = shutOutIsHigher + ? await StartPair(quick, never) + : await StartPair(never, quick); + var (survivor, shutOut) = shutOutIsHigher ? (lower, higher) : (higher, lower); + + Assert.True(await Throttle(lower, higher, Blackhole.Instance)); + await TwoNodeClusterFixture.WaitForMemberRemoved(survivor, Akka.Cluster.Cluster.Get(shutOut).SelfAddress, TimeSpan.FromSeconds(30)); + Assert.Contains(Akka.Cluster.Cluster.Get(shutOut).State.Members, + m => m.Address == Akka.Cluster.Cluster.Get(survivor).SelfAddress); // it still counts the survivor + Assert.True(await Throttle(lower, higher, Unthrottled.Instance)); + + Assert.True(await Terminates(shutOut, TimeSpan.FromSeconds(30)), + "the shut-out node kept running as a second active node"); + Assert.False(await Terminates(survivor, TimeSpan.FromSeconds(5)), "the survivor restarted too"); + Assert.Equal(MemberStatus.Up, Akka.Cluster.Cluster.Get(survivor).SelfMember.Status); + } + + /// + /// A partition past the window, then healed: both sides downed and removed each other, so both + /// are refused — exactly ONE restarts, the higher address. A fresh incarnation on the higher + /// port (what the service supervisor starts) rejoins the survivor, and the survivor is still Up + /// after its grace has run out. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task HealedPartition_BothShutOut_OnlyTheHigherAddressRestarts_AndRejoins() + { + var stable = TimeSpan.FromSeconds(2); + var (lower, higher, lowerPort, higherPort) = await StartPair(stable, stable); + var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress; + var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress; + + Assert.True(await Throttle(lower, higher, Blackhole.Instance)); + await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30)); + Assert.False(lower.WhenTerminated.IsCompleted || higher.WhenTerminated.IsCompleted, + "a node restarted while the partition was still in place"); + Assert.True(await Throttle(lower, higher, Unthrottled.Instance)); + + Assert.True(await Terminates(higher, TimeSpan.FromSeconds(30)), + "the higher-address node did not restart after the partition healed"); + Assert.False(lower.WhenTerminated.IsCompleted, "the lower-address node restarted as well"); + + // The supervisor's restart: a fresh incarnation on the same address rejoins the survivor. + var reborn = StartNode(higherPort, lowerPort, stable); + await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(60)); + await TwoNodeClusterFixture.WaitForMembersUp(reborn, 2, TimeSpan.FromSeconds(60)); + + // Past the survivor's 20s grace: it saw the peer rejoin, so it stays. + Assert.False(await Terminates(lower, TimeSpan.FromSeconds(25)), "the survivor restarted after its peer rejoined"); + Assert.Equal(2, Akka.Cluster.Cluster.Get(lower).State.Members.Count(m => m.Status == MemberStatus.Up)); + Assert.Empty(Akka.Cluster.Cluster.Get(lower).State.Unreachable); + } + + /// + /// Switch OFF: the same healed partition keeps BOTH sides running (the pre-#132 dual-active) + /// — the positive control that the restarts above come from the feature. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task HealedPartition_SwitchOff_BothStayUp() + { + var stable = TimeSpan.FromSeconds(2); + var (lower, higher, _, _) = await StartPair(stable, stable, restartWhenShutOut: false); + var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress; + var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress; + + Assert.True(await Throttle(lower, higher, Blackhole.Instance)); + await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30)); + Assert.True(await Throttle(lower, higher, Unthrottled.Instance)); + + Assert.False(await Terminates(higher, TimeSpan.FromSeconds(15)), "a node restarted with the switch off"); + Assert.False(lower.WhenTerminated.IsCompleted, "a node restarted with the switch off"); + } + + /// Prepares the fixture before the tests run. + /// A task that represents the asynchronous operation. + public Task InitializeAsync() => Task.CompletedTask; + + /// Releases the resources held by this fixture asynchronously. + /// A task that represents the asynchronous operation. + public async Task DisposeAsync() + { + foreach (var s in _systems) + { + try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ } + } + } +}