fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132)
A node frozen past failure detection + StableAfter, or both sides of a partition that heals after each downed the other, used to stay a second active node until an operator restarted one: the peer quarantines the removed UID and Akka.Cluster 1.5.62 never acts on ThisActorSystemQuarantinedEvent. ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for leader-auto-down only) watches two signals: - S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer is still a live member here, only this side can be shut out -> restart now. - S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still runs the incarnation this node removed. A healed partition yields only S2, on both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then tiebreak. Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower stays and restarts only if still split one grace later. Removed peers are probed (Identify, 5s -> 60s back-off) since nothing else contacts them after mutual removal. Restart = Cluster.Down(self) -> existing self-down path. The Host now exits 75 after an unplanned ActorSystem termination (and sets it on the Windows service before it reports stopped): SCM recovery with failureflag 1 only restarts a service that stops with a non-zero code, and the watchdog used to exit 0. Shipped default strategy stays auto-down. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -111,7 +111,9 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
### Downed-node recovery
|
||||
|
||||
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
|
||||
`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits **with code 75** (`AkkaHostedService.SelfDownExitCode`; set by `Program` after the host stops, and on the Windows service itself before it reports `SERVICE_STOPPED`) and the service supervisor (docker `restart: unless-stopped`, Windows service recovery with `failureflag 1`, which only acts on a non-zero exit code) restarts it as a fresh incarnation (`AkkaHostedService.GetOrCreateActorSystem`).
|
||||
|
||||
Under `leader-auto-down` the same path restarts a node that its peer has **shut out** — downed, removed and quarantined it (a frozen node coming back), or both sides of a healed partition, where the higher address restarts. See `Component-ClusterInfrastructure.md` → *Shut-out restart* (ScadaBridge#132).
|
||||
|
||||
**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
|
||||
|
||||
|
||||
@@ -104,3 +104,36 @@ Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the prove
|
||||
- **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that
|
||||
both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged).
|
||||
- **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs.
|
||||
|
||||
## 7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132)
|
||||
|
||||
The #116 rig drill found a gap that `AutoDowning` and its port share: a node frozen past failure detection plus the window
|
||||
comes back as a **permanent second active node**, and the same happens to both sides of a partition that heals after each
|
||||
side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote
|
||||
publishes `ThisActorSystemQuarantinedEvent` on the stale node, and Akka.Cluster never subscribes to it.
|
||||
|
||||
**Owner decision (#132):** under `leader-auto-down`, a node that is shut out restarts itself; when both sides are shut out,
|
||||
exactly one restarts, by a deterministic tiebreak. Implemented as `ShutOutRestart` (Host, a child of the downing actor, so
|
||||
`LeaderAutoDownBase` stays a line-for-line port), behind `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default
|
||||
`true` for this strategy only) and `ShutOutGrace` (default 60 s). The shipped default strategy stays `auto-down`.
|
||||
|
||||
- **Two signals, confirmed in-process.** S1 `ThisActorSystemQuarantinedEvent` (the frozen node). S2 an `AssociationErrorEvent`
|
||||
whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed
|
||||
partition produces **only S2, on both sides** — each side aborts the other's handshake with its own refused UID before a
|
||||
quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with
|
||||
`Identify` (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it.
|
||||
- **Decision.** S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this
|
||||
node removed the peer: tiebreak. S2: wait `ShutOutGrace` (a frozen peer restarts itself meanwhile), then tiebreak if still
|
||||
split. Tiebreak: **higher address restarts**, the lower stays and restarts only if the peer is still split from it one
|
||||
more grace later.
|
||||
- **Why address, not age:** see `Component-ClusterInfrastructure.md` → *Shut-out restart*. In short, both nodes hold both
|
||||
addresses always; the peer's up-number is history that can differ after the split.
|
||||
- **Exit code.** The restart is the existing self-down path, and it exposed a latent gap: the watchdog's `StopApplication()`
|
||||
exited 0, and Windows service recovery with `failureflag 1` acts only on a non-zero exit. The Host now exits 75 after an
|
||||
unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code.
|
||||
- **Contract change for #116 gate 4:** the partition still produces dual-active *while it lasts*; after it heals, one side
|
||||
(the higher address) restarts and the pair converges with no operator action.
|
||||
`LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds (it never heals).
|
||||
- **Tests:** `ShutOutRestartTests` (unit), `ShutOutRestartScenarioTests` (two real nodes: freeze shape both address orders,
|
||||
healed partition, switch off), `UnexpectedTerminationTests`, `HoconBuilderTests`, `ClusterOptionsValidatorTests`.
|
||||
Rig evidence: `2026-11-12-leader-auto-down-rig-drill.md` §7.
|
||||
|
||||
@@ -104,6 +104,24 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
|
||||
|
||||
Without it, a node that was **frozen** (VM pause, long GC stall, `docker pause`) for longer than failure detection plus `StableAfter` comes back as a **second, permanent active node**: its peer has already downed and removed it, quarantining its UID, and Akka.Cluster 1.5.62 never tells the frozen node — so it downs the peer in its own one-node view. Both sides of a **real partition that heals** after each side downed the other end the same way. This happens under `auto-down` too (Akka core behaviour); the in-repo provider is where it is fixed.
|
||||
|
||||
With `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` on (**the default whenever `leader-auto-down` is selected**), the `ShutOutRestart` actor (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs`, a child of the downing actor) restarts the stale side:
|
||||
|
||||
| Signal (Akka.NET 1.5.62) | Who sees it | Decision |
|
||||
|---|---|---|
|
||||
| **S1** `ThisActorSystemQuarantinedEvent` — the peer has quarantined *this* node — while the peer is still a live member in this node's view | the frozen node | **restart now**; no tiebreak (this node has not removed the peer, so the peer cannot be shut out by it) |
|
||||
| **S1** after this node has removed the peer too | either side of a split | **tiebreak now** |
|
||||
| **S2** `AssociationErrorEvent` "remote system has a UID that has been quarantined" — the peer still runs the incarnation *this* node removed | both sides of a healed partition; the survivor of a freeze | wait `ShutOutGrace` (a frozen peer restarts itself on S1 meanwhile); still split at the end ⇒ **tiebreak** |
|
||||
|
||||
**Tiebreak: the higher address (host, then port — Akka's `Member.AddressOrdering`) restarts; the lower stays.** The addresses are in every signal and in both nodes' seed lists, so both nodes compute the same answer without talking. Member age (up-number) was rejected: by the time a signal fires both sides have removed each other, so the peer's up-number exists only as remembered history that can differ (an unconverged join, a restart since), and a disagreement means both restart or neither. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. The lower node gives its peer one more `ShutOutGrace` and restarts only if the peer has neither rejoined nor gone quiet by then (the single-sided case where the stale side happened to be the lower address).
|
||||
|
||||
After both sides have removed each other neither sends the other anything, so a healed partition would stay silent: the actor **probes** (an `Identify` to `/system/cluster/core/daemon`) every peer it saw removed after a Down, backing off 5 s → 60 s, until that address is a member again. Each probe of a dead peer logs one Akka association warning.
|
||||
|
||||
"Restart" is the existing self-down path (below): `Cluster.Down(self)` ⇒ the `ActorSystem` terminates ⇒ the process exits **75** ⇒ the supervisor restarts it and it rejoins as standby. Every decision logs a `[leader-auto-down]` line: `shut out by <peer> … tiebreak not needed — this node RESTARTS`, `split from <peer> … Giving it <grace> …`, `Tiebreak says RESTART|STAY`, `rejoined … stays`. Settings: `RestartWhenShutOut` (bool, default `true`) and `ShutOutGrace` (default `60s`, must be positive, must cover the peer's worst-case shutdown). Off restores the pre-#132 behaviour (dual-active until an operator restarts one side). Proven by `ShutOutRestartTests` (unit) and `ShutOutRestartScenarioTests` (real two-node clusters, freeze shape and healed partition); rig evidence in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7.
|
||||
|
||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||
|
||||
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
|
||||
@@ -116,8 +134,8 @@ Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `
|
||||
When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
|
||||
|
||||
1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
|
||||
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
|
||||
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
|
||||
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits — with code 75** (`AkkaHostedService.SelfDownExitCode`, `EX_TEMPFAIL`; ScadaBridge#132). Under the Windows service lifetime the code is also set on the service (`ServiceBase.ExitCode`) before the stop, so the SCM sees it.
|
||||
3. The service supervisor restarts it — docker `restart: unless-stopped` (any exit code), or Windows service recovery actions (`sc.exe failure … restart/…` **plus `sc.exe failureflag <svc> 1`**, both set by the wonder install script `deploy/wonder-app-vd03/install.ps1`, which is kept outside git). With `failureflag 1` the SCM runs the recovery actions for a service that stops with a **non-zero** exit code; before #132 the watchdog exit reported 0, which the SCM treats as a clean stop. (ScadaBridge runs as native .NET Windows services; NSSM wraps only the LDAP directory.)
|
||||
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
|
||||
|
||||
### Seed Node Ordering
|
||||
|
||||
@@ -144,6 +144,61 @@ public class ClusterOptions
|
||||
/// <c>docs/plans/2026-11-09-tls-production-profile-design.md</c>.
|
||||
/// </summary>
|
||||
public ClusterRemotingTlsOptions RemotingTls { get; set; } = new();
|
||||
|
||||
/// <summary>
|
||||
/// Settings that apply only when <see cref="SplitBrainResolverStrategy"/> is
|
||||
/// <c>leader-auto-down</c> (<c>ScadaBridge:Cluster:LeaderAutoDown</c>, ScadaBridge#132).
|
||||
/// Inert under every other strategy: <c>AkkaHostedService.BuildHocon</c> emits none of it.
|
||||
/// </summary>
|
||||
public LeaderAutoDownOptions LeaderAutoDown { get; set; } = new();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>leader-auto-down</c>-only settings (ScadaBridge#132): what a node does when it learns that
|
||||
/// its pair peer has <b>shut it out</b> — removed and quarantined it, so it can never talk to that
|
||||
/// peer again until it restarts.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why it exists.</b> A node that was frozen (VM pause, GC stall, <c>docker pause</c>) past the
|
||||
/// failure-detection window plus <see cref="ClusterOptions.StableAfter"/> is downed and removed by
|
||||
/// its peer. When it resumes, every association is refused with "the remote system has quarantined
|
||||
/// this system" — Akka.Remote publishes <c>ThisActorSystemQuarantinedEvent</c>, but Akka.Cluster
|
||||
/// 1.5.62 never acts on it, so the frozen node downs the peer in its own one-node view and becomes
|
||||
/// a second, permanent active node. The same thing happens to <b>both</b> nodes after a real
|
||||
/// network partition heals.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>What it does</b> (with <see cref="RestartWhenShutOut"/> on): a node that is shut out by its
|
||||
/// peer while it still counts that peer as a member restarts itself (it is the stale side). When it
|
||||
/// has already removed the peer too — both sides shut each other out — a deterministic tiebreak
|
||||
/// both nodes compute identically picks ONE side: the node with the <b>higher</b> address
|
||||
/// (host, then port; Akka's own member address ordering) restarts, the lower stays. The lower node
|
||||
/// restarts too only if it is still being shut out after <see cref="ShutOutGrace"/>. "Restart" is
|
||||
/// the existing self-down path: <c>Cluster.Down(self)</c>, the <c>ActorSystem</c> terminates, the
|
||||
/// Host exits non-zero and the service supervisor restarts it as a fresh incarnation that rejoins
|
||||
/// as standby. Implemented by <c>ShutOutRestart</c> in the Host.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class LeaderAutoDownOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Whether a node shut out by its pair peer restarts itself (see the class remarks). Default
|
||||
/// <see langword="true"/> — on whenever <c>leader-auto-down</c> is selected. Set
|
||||
/// <see langword="false"/> to get the old behaviour back: a frozen node or a healed partition
|
||||
/// leaves two active nodes until an operator restarts one.
|
||||
/// </summary>
|
||||
public bool RestartWhenShutOut { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// How long the tiebreak WINNER (the lower-address node) waits for its peer to restart before
|
||||
/// it re-checks. At the end of the grace it restarts only if the peer is still not a member AND
|
||||
/// it was shut out again in the last few seconds — i.e. the peer never restarted, so this node
|
||||
/// is the stale side after all. Must cover the peer's worst-case shut-down time (its old
|
||||
/// incarnation keeps refusing until its <c>ActorSystem</c> has terminated). Default 60s.
|
||||
/// Validated <c>> 0</c> at boot when the feature is on.
|
||||
/// </summary>
|
||||
public TimeSpan ShutOutGrace { get; set; } = TimeSpan.FromSeconds(60);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -74,6 +74,17 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
|
||||
"declared unreachable before a heartbeat can arrive.");
|
||||
|
||||
// ScadaBridge#132: the shut-out restart grace is only consulted under leader-auto-down
|
||||
// with the feature on; a zero/negative grace would make the tiebreak winner restart on
|
||||
// the spot, i.e. BOTH nodes of a healed partition restart together.
|
||||
var isLeaderAutoDown = string.Equals(
|
||||
options.SplitBrainResolverStrategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase);
|
||||
var leaderAutoDown = options.LeaderAutoDown ?? new LeaderAutoDownOptions();
|
||||
builder.RequireThat(
|
||||
!isLeaderAutoDown || !leaderAutoDown.RestartWhenShutOut || leaderAutoDown.ShutOutGrace > TimeSpan.Zero,
|
||||
"ClusterOptions.LeaderAutoDown.ShutOutGrace must be a positive duration when "
|
||||
+ "LeaderAutoDown.RestartWhenShutOut is on (ScadaBridge#132).");
|
||||
|
||||
// DownIfAlone is a keep-oldest knob; under (leader-)auto-down each side downs the
|
||||
// unreachable peer regardless, so the flag is inert and any value is fine.
|
||||
var isKeepOldest = string.Equals(
|
||||
|
||||
@@ -34,8 +34,20 @@ public class AkkaHostedService : IHostedService
|
||||
private readonly CommunicationOptions _communicationOptions;
|
||||
private readonly ILogger<AkkaHostedService> _logger;
|
||||
private readonly IHostApplicationLifetime? _appLifetime;
|
||||
private readonly IHostLifetime? _hostLifetime;
|
||||
private ActorSystem? _actorSystem;
|
||||
|
||||
/// <summary>
|
||||
/// Process exit code after an ActorSystem termination outside host shutdown (a self-down,
|
||||
/// or a ScadaBridge#132 shut-out restart): 75, <c>EX_TEMPFAIL</c> — "temporary failure, try
|
||||
/// again". It must be NON-zero: docker <c>restart: unless-stopped</c> restarts on any exit, but
|
||||
/// Windows service recovery (<c>sc.exe failure … restart</c> with <c>failureflag 1</c>, set by
|
||||
/// <c>deploy/wonder-app-vd03/install.ps1</c>) applies its restart actions to a service that
|
||||
/// reports <c>SERVICE_STOPPED</c> only when the reported exit code is not 0. A clean
|
||||
/// <c>StopApplication()</c> alone reports 0, so the node would stay stopped.
|
||||
/// </summary>
|
||||
public const int SelfDownExitCode = 75;
|
||||
|
||||
/// <summary>
|
||||
/// Flips to <c>true</c> the moment <see cref="StopAsync"/> begins, so the
|
||||
/// <see cref="ActorSystem.WhenTerminated"/> watchdog can tell an
|
||||
@@ -84,14 +96,17 @@ public class AkkaHostedService : IHostedService
|
||||
/// <param name="communicationOptions">The communication configuration options.</param>
|
||||
/// <param name="logger">The logger instance.</param>
|
||||
/// <param name="appLifetime">The host application lifetime; optional so existing construction sites (tests, DI variants) keep working without it.</param>
|
||||
/// <param name="hostLifetime">The host lifetime; when it is the Windows service lifetime, the self-down exit code is reported to the SCM through it. Optional.</param>
|
||||
public AkkaHostedService(
|
||||
IServiceProvider serviceProvider,
|
||||
IOptions<NodeOptions> nodeOptions,
|
||||
IOptions<ClusterOptions> clusterOptions,
|
||||
IOptions<CommunicationOptions> communicationOptions,
|
||||
ILogger<AkkaHostedService> logger,
|
||||
IHostApplicationLifetime? appLifetime = null)
|
||||
IHostApplicationLifetime? appLifetime = null,
|
||||
IHostLifetime? hostLifetime = null)
|
||||
{
|
||||
_hostLifetime = hostLifetime;
|
||||
_serviceProvider = serviceProvider;
|
||||
_nodeOptions = nodeOptions.Value;
|
||||
_clusterOptions = clusterOptions.Value;
|
||||
@@ -212,8 +227,11 @@ public class AkkaHostedService : IHostedService
|
||||
{
|
||||
if (_stopRequested) return;
|
||||
_logger.LogCritical(
|
||||
"ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
|
||||
+ "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
|
||||
"ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / "
|
||||
+ "leader-auto-down shut-out restart). Stopping the host process with exit code {ExitCode} so the "
|
||||
+ "service supervisor restarts this node as a fresh incarnation.", SelfDownExitCode);
|
||||
_terminatedOutsideHostShutdown = true;
|
||||
MarkServiceSelfDownExit(_hostLifetime);
|
||||
_appLifetime?.StopApplication();
|
||||
}, TaskContinuationOptions.ExecuteSynchronously);
|
||||
|
||||
@@ -223,6 +241,37 @@ public class AkkaHostedService : IHostedService
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <see langword="true"/> once the <see cref="ActorSystem"/> has terminated outside
|
||||
/// <see cref="StopAsync"/> (a self-down or a shut-out restart). <c>Program</c> reads it after
|
||||
/// <c>RunAsync</c> returns and exits with <see cref="SelfDownExitCode"/>; it is a flag rather than a
|
||||
/// direct write of the process exit code so test hosts that terminate a system never inherit
|
||||
/// a non-zero process exit code.
|
||||
/// </summary>
|
||||
public bool TerminatedOutsideHostShutdown => _terminatedOutsideHostShutdown;
|
||||
|
||||
private volatile bool _terminatedOutsideHostShutdown;
|
||||
|
||||
/// <summary>
|
||||
/// When the host runs as a Windows service, sets <see cref="SelfDownExitCode"/> as the exit code
|
||||
/// the service reports to the SCM when it stops — the value Windows service recovery
|
||||
/// (<c>failureflag 1</c>) keys its restart on. Must run BEFORE <c>StopApplication()</c>: the
|
||||
/// Windows service lifetime reports <c>SERVICE_STOPPED</c> during host shutdown, before
|
||||
/// <c>Program</c> can set the process exit code.
|
||||
/// </summary>
|
||||
/// <param name="hostLifetime">The host lifetime; the Windows service lifetime is a <c>ServiceBase</c>.</param>
|
||||
/// <returns><see langword="true"/> when the exit code was set on a Windows service lifetime.</returns>
|
||||
internal static bool MarkServiceSelfDownExit(IHostLifetime? hostLifetime)
|
||||
{
|
||||
if (OperatingSystem.IsWindows() && hostLifetime is System.ServiceProcess.ServiceBase service)
|
||||
{
|
||||
service.ExitCode = SelfDownExitCode;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the Akka HOCON configuration document. Every interpolated value is
|
||||
/// routed through <see cref="QuoteHocon"/> (string values) so a hostname,
|
||||
@@ -313,8 +362,12 @@ public class AkkaHostedService : IHostedService
|
||||
// The leader-auto-down duration sits outside akka.cluster so Akka never mistakes it
|
||||
// for one of its own (deprecated) auto-down keys. Only emitted for that strategy, so
|
||||
// the auto-down / keep-oldest documents are byte-identical to before.
|
||||
// ScadaBridge#132: the shut-out restart switch and grace ride in the same block.
|
||||
var shutOut = clusterOptions.LeaderAutoDown ?? new LeaderAutoDownOptions();
|
||||
var leaderAutoDownBlock = string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase)
|
||||
? $"{LeaderAutoDowningProvider.UnreachableAfterPath} = {DurationHocon(clusterOptions.StableAfter)}\n"
|
||||
+ $"{ShutOutRestartSettings.EnabledPath} = {(shutOut.RestartWhenShutOut ? "on" : "off")}\n"
|
||||
+ $"{ShutOutRestartSettings.GracePath} = {DurationHocon(shutOut.ShutOutGrace)}\n"
|
||||
: string.Empty;
|
||||
|
||||
// Akka.Remote TLS (ScadaBridge#64, opt-in). Emitted inside dot-netty.tcp only when
|
||||
|
||||
@@ -88,7 +88,8 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider
|
||||
$"LeaderAutoDowningProvider selected but '{UnreachableAfterPath}' not set");
|
||||
}
|
||||
|
||||
return LeaderAutoDown.Props(after.Value, _cluster);
|
||||
return LeaderAutoDown.Props(
|
||||
after.Value, _cluster, ShutOutRestartSettings.FromConfig(_system.Settings.Config));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -118,21 +119,29 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider
|
||||
/// </summary>
|
||||
internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||
{
|
||||
/// <summary>Child name of the <see cref="ShutOutRestart"/> actor.</summary>
|
||||
internal const string ShutOutRestartName = "shut-out-restart";
|
||||
|
||||
private readonly Akka.Cluster.Cluster _cluster;
|
||||
private readonly ShutOutRestartSettings? _shutOut;
|
||||
|
||||
/// <summary>Props for the downing actor.</summary>
|
||||
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||
/// <param name="cluster">The cluster extension.</param>
|
||||
public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) =>
|
||||
Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster));
|
||||
/// <param name="shutOut">Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling.</param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut = null) =>
|
||||
Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster, shutOut));
|
||||
|
||||
/// <summary>Creates the actor.</summary>
|
||||
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||
/// <param name="cluster">The cluster extension.</param>
|
||||
public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster)
|
||||
/// <param name="shutOut">Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling.</param>
|
||||
public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut)
|
||||
: base(unreachableAfter)
|
||||
{
|
||||
_cluster = cluster;
|
||||
_shutOut = shutOut;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
@@ -145,6 +154,22 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||
protected override void PreStart()
|
||||
{
|
||||
_cluster.Subscribe(Self, typeof(ClusterEvent.IClusterDomainEvent));
|
||||
|
||||
// ScadaBridge#132: the shut-out restart is a separate child so the downing logic above
|
||||
// stays a line-for-line port of Akka's AutoDownBase. A child, not a sibling, so it lives
|
||||
// and dies with this provider's actor and exists only under leader-auto-down.
|
||||
if (_shutOut is { Enabled: true })
|
||||
{
|
||||
Context.ActorOf(ShutOutRestart.Props(_shutOut, _cluster), ShutOutRestartName);
|
||||
}
|
||||
else
|
||||
{
|
||||
Context.GetLogger().Info(
|
||||
"[leader-auto-down] shut-out restart is OFF for {0} (ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut = false): "
|
||||
+ "a node its peer has removed and quarantined keeps running as a second active node until it is restarted",
|
||||
_cluster.SelfAddress);
|
||||
}
|
||||
|
||||
base.PreStart();
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,674 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Akka.Event;
|
||||
using Akka.Remote;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// Settings for <see cref="ShutOutRestart"/>, read from the <c>scadabridge.cluster.leader-auto-down</c>
|
||||
/// HOCON block that <see cref="AkkaHostedService.BuildHocon"/> emits from
|
||||
/// <c>ScadaBridge:Cluster:LeaderAutoDown</c> (ScadaBridge#132).
|
||||
/// </summary>
|
||||
/// <param name="Enabled">Whether a node that is shut out by its peer restarts itself.</param>
|
||||
/// <param name="Grace">How long a peer gets to restart itself before this node acts (see <see cref="ShutOutRestart"/>).</param>
|
||||
/// <param name="ProbeInterval">First interval between probes of a removed peer, and the fixed interval while a grace runs.</param>
|
||||
/// <param name="ProbeMaxInterval">Cap for the probe back-off while no grace runs.</param>
|
||||
internal sealed record ShutOutRestartSettings(
|
||||
bool Enabled, TimeSpan Grace, TimeSpan ProbeInterval, TimeSpan ProbeMaxInterval)
|
||||
{
|
||||
/// <summary>HOCON switch: <c>on</c>/<c>off</c>. Missing = on.</summary>
|
||||
public const string EnabledPath = "scadabridge.cluster.leader-auto-down.restart-when-shut-out";
|
||||
|
||||
/// <summary>HOCON duration for <see cref="Grace"/>. Missing = 60s.</summary>
|
||||
public const string GracePath = "scadabridge.cluster.leader-auto-down.shut-out-grace";
|
||||
|
||||
/// <summary>HOCON duration for <see cref="ProbeInterval"/>. Missing = 5s. Not bound from appsettings (tests only).</summary>
|
||||
public const string ProbeIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-interval";
|
||||
|
||||
/// <summary>HOCON duration for <see cref="ProbeMaxInterval"/>. Missing = 60s. Not bound from appsettings (tests only).</summary>
|
||||
public const string ProbeMaxIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval";
|
||||
|
||||
/// <summary>
|
||||
/// How recent the last shut-out evidence must be, when a grace ends, for the split to count as
|
||||
/// still in place: three probe intervals, so a missed probe or two does not flip the verdict.
|
||||
/// </summary>
|
||||
public TimeSpan RecentWindow => ProbeInterval * 3;
|
||||
|
||||
/// <summary>Reads the settings, defaulting every missing key.</summary>
|
||||
/// <param name="config">The actor system's configuration.</param>
|
||||
/// <returns>The settings.</returns>
|
||||
public static ShutOutRestartSettings FromConfig(Config config)
|
||||
{
|
||||
var enabled = !config.HasPath(EnabledPath) || config.GetBoolean(EnabledPath, true);
|
||||
return new ShutOutRestartSettings(
|
||||
enabled,
|
||||
Duration(config, GracePath, TimeSpan.FromSeconds(60)),
|
||||
Duration(config, ProbeIntervalPath, TimeSpan.FromSeconds(5)),
|
||||
Duration(config, ProbeMaxIntervalPath, TimeSpan.FromSeconds(60)));
|
||||
}
|
||||
|
||||
private static TimeSpan Duration(Config config, string path, TimeSpan fallback) =>
|
||||
config.HasPath(path) ? config.GetTimeSpan(path, fallback, true) : fallback;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132: makes a node that is split from its pair peer — the peer has downed and removed
|
||||
/// it, or each has removed the other — restart itself, so a frozen node or a healed partition does
|
||||
/// not leave two active nodes until an operator acts. Runs as a child of the
|
||||
/// <see cref="LeaderAutoDown"/> downing actor, so it exists only under <c>leader-auto-down</c>, and
|
||||
/// only when <c>ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut</c> is on (the default for
|
||||
/// that strategy).
|
||||
/// <para>
|
||||
/// <b>The signals</b> (Akka.NET 1.5.62, read from the decompiled assemblies and confirmed by the
|
||||
/// in-process scenario tests). When a node removes a downed member, its cluster quarantines that
|
||||
/// member's UID. After that, each association between the two incarnations fails, and which side
|
||||
/// learns what depends on who still counts whom as a member:
|
||||
/// <list type="bullet">
|
||||
/// <item><b>S1 — "the peer has quarantined THIS node".</b> Akka.Remote publishes
|
||||
/// <see cref="ThisActorSystemQuarantinedEvent"/> (<c>EndpointManager</c>, on an
|
||||
/// <c>InvalidAssociation</c> whose disassociate reason is <c>Quarantined</c>). Seen by a node that
|
||||
/// still talks to a peer which has removed it: the frozen node after <c>docker pause</c>. Akka.Cluster
|
||||
/// 1.5.62 never subscribes to it — <c>ClusterCoreDaemon</c> handles only the opposite
|
||||
/// <c>QuarantinedEvent</c> — which is the root cause of #132.</item>
|
||||
/// <item><b>S2 — "the peer is still running the incarnation THIS node removed".</b> An outbound
|
||||
/// handshake that meets a UID this node has quarantined is aborted (<c>ProtocolStateActor</c>,
|
||||
/// <c>ForbiddenUidReason</c>: "The remote system has a UID that has been quarantined"), surfacing as
|
||||
/// an <see cref="AssociationErrorEvent"/>. Seen by both sides of a partition that healed after each
|
||||
/// side downed the other — neither of them sees S1, because each aborts the other's handshake first —
|
||||
/// and by the survivor of a freeze when it probes the frozen node.</item>
|
||||
/// </list>
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The decision.</b>
|
||||
/// <list type="number">
|
||||
/// <item><b>S1 while the peer is still a live member in this node's view</b> (Joining, WeaklyUp, Up
|
||||
/// or Leaving): this node has not removed the peer, so it has not quarantined it, so the peer
|
||||
/// cannot be shut out by this node — only this side is. This is the frozen node. <b>Restart
|
||||
/// now</b>; no tiebreak is needed.</item>
|
||||
/// <item><b>S1 after this node has removed the peer too</b>: both sides are shut out.
|
||||
/// <b>Tiebreak now</b> (below).</item>
|
||||
/// <item><b>S2</b>: the peer's old incarnation is alive. It may be a frozen node about to restart
|
||||
/// itself on S1 — so this node first gives it <see cref="ShutOutRestartSettings.Grace"/>. If, at
|
||||
/// the end, the peer has neither rejoined nor gone quiet, both sides are shut out: <b>tiebreak</b>.
|
||||
/// </item>
|
||||
/// <item><b>Tiebreak</b>: the node with the <b>higher</b> address restarts now. The lower one stays
|
||||
/// and gives the peer one more <see cref="ShutOutRestartSettings.Grace"/>; it restarts only if the
|
||||
/// peer has still not rejoined and is still producing evidence at the end — the single-sided case
|
||||
/// where the stale side happened to be the lower address.</item>
|
||||
/// </list>
|
||||
/// "Quiet" means no S1/S2 from that peer for <see cref="ShutOutRestartSettings.RecentWindow"/>: the
|
||||
/// old incarnation is gone (restarting, or dead), so there is nothing left to split from.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why the address tiebreak, not member age.</b> Both nodes must reach the same answer without
|
||||
/// talking to each other, from state they already hold. The two addresses are in every signal and
|
||||
/// in both nodes' seed lists, so the comparison is identical on both sides by construction. Age
|
||||
/// (up-number) is not: by the time a signal fires both sides have removed each other, so the peer's
|
||||
/// up-number survives only as remembered history, which differs if a join never converged before
|
||||
/// the split or either side has restarted since. A disagreement there means both restart or neither
|
||||
/// does — exactly what the tiebreak exists to prevent. The ordering is Akka's own
|
||||
/// <see cref="Member.AddressOrdering"/> (host, then port), which also picks the Akka leader of the
|
||||
/// re-formed pair, and matches the bootstrap guard, where the lower address is the preferred founder.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Probing.</b> After both sides have removed each other neither sends the other anything, so a
|
||||
/// healed partition would stay silent and neither signal would ever fire. This actor therefore
|
||||
/// sends an <see cref="Identify"/> to every peer it saw removed after being downed, with a back-off
|
||||
/// from <see cref="ShutOutRestartSettings.ProbeInterval"/> to
|
||||
/// <see cref="ShutOutRestartSettings.ProbeMaxInterval"/> (fixed at the short interval while a grace
|
||||
/// runs), until that address is a member again. A probe of a restarted incarnation (new UID) is
|
||||
/// harmless — the new incarnation is joining anyway; a probe of the old incarnation produces S2
|
||||
/// here and S1 or S2 there.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>The restart</b> reuses the existing self-down path: <c>Cluster.Down(self)</c> ⇒
|
||||
/// <c>run-coordinated-shutdown-when-down</c> ⇒ the <c>ActorSystem</c> terminates ⇒ the Host's
|
||||
/// <c>WhenTerminated</c> watchdog exits the process with <see cref="AkkaHostedService.SelfDownExitCode"/>
|
||||
/// ⇒ the service supervisor (docker <c>restart: unless-stopped</c>, Windows service recovery with
|
||||
/// <c>failureflag 1</c>) starts a fresh incarnation that rejoins its peer as standby. If the system
|
||||
/// has not terminated <see cref="FallbackShutdownAfter"/> after the Down, <c>CoordinatedShutdown</c>
|
||||
/// is run directly.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class ShutOutRestart : ShutOutRestartBase
|
||||
{
|
||||
/// <summary>How long after <c>Cluster.Down(self)</c> the direct <c>CoordinatedShutdown</c> fallback runs.</summary>
|
||||
internal static readonly TimeSpan FallbackShutdownAfter = TimeSpan.FromSeconds(20);
|
||||
|
||||
private readonly Akka.Cluster.Cluster _cluster;
|
||||
private ICancelable? _fallback;
|
||||
|
||||
/// <summary>Props for the actor.</summary>
|
||||
/// <param name="settings">The settings.</param>
|
||||
/// <param name="cluster">The cluster extension.</param>
|
||||
/// <returns>The props.</returns>
|
||||
public static Props Props(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster) =>
|
||||
Akka.Actor.Props.Create(() => new ShutOutRestart(settings, cluster));
|
||||
|
||||
/// <summary>Creates the actor.</summary>
|
||||
/// <param name="settings">The settings.</param>
|
||||
/// <param name="cluster">The cluster extension.</param>
|
||||
public ShutOutRestart(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster)
|
||||
: base(settings)
|
||||
{
|
||||
_cluster = cluster;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Address SelfAddress => _cluster.SelfAddress;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override IScheduler Scheduler => Context.System.Scheduler;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override IEnumerable<Address> SeedNodes => _cluster.Settings.SeedNodes;
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
_cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot,
|
||||
typeof(ClusterEvent.IMemberEvent));
|
||||
Context.System.EventStream.Subscribe(Self, typeof(ThisActorSystemQuarantinedEvent));
|
||||
Context.System.EventStream.Subscribe(Self, typeof(AssociationErrorEvent));
|
||||
base.PreStart();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
_cluster.Unsubscribe(Self);
|
||||
Context.System.EventStream.Unsubscribe(Self);
|
||||
_fallback?.Cancel();
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void RestartSelf()
|
||||
{
|
||||
_cluster.Down(_cluster.SelfAddress);
|
||||
_fallback = Scheduler.ScheduleTellOnceCancelable(FallbackShutdownAfter, Self, FallbackShutdown.Instance, Self);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void Probe(Address peer)
|
||||
{
|
||||
Context.ActorSelection(new RootActorPath(peer) / "system" / "cluster" / "core" / "daemon")
|
||||
.Tell(new Identify(ProbeMessageId), Self);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void OnFallbackShutdown()
|
||||
{
|
||||
if (!Context.System.WhenTerminated.IsCompleted)
|
||||
{
|
||||
Log.Warning(
|
||||
"[leader-auto-down] {0} was downed for a shut-out restart {1} ago but its actor system is still running; running CoordinatedShutdown directly",
|
||||
SelfAddress, FallbackShutdownAfter);
|
||||
CoordinatedShutdown.Get(Context.System).Run(CoordinatedShutdown.ClusterDowningReason.Instance);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The cluster-free decision logic of <see cref="ShutOutRestart"/>, split out (as
|
||||
/// <see cref="LeaderAutoDownBase"/> is) so the tiebreak and the single- and both-sides cases can be
|
||||
/// unit tested by feeding events straight to the actor. Every decision is logged with the
|
||||
/// <c>[leader-auto-down]</c> prefix; the restart / stay verdicts at Warning.
|
||||
/// </summary>
|
||||
internal abstract class ShutOutRestartBase : UntypedActor
|
||||
{
|
||||
/// <summary>The <see cref="Identify.MessageId"/> carried by every probe.</summary>
|
||||
internal const string ProbeMessageId = "scadabridge-shut-out-probe";
|
||||
|
||||
/// <summary>Timer: probe <see cref="Peer"/> again.</summary>
|
||||
/// <param name="Peer">The removed peer to probe.</param>
|
||||
internal sealed record ProbeTick(Address Peer);
|
||||
|
||||
/// <summary>Timer: the grace for <see cref="Peer"/> has ended.</summary>
|
||||
/// <param name="Peer">The peer this node is split from.</param>
|
||||
internal sealed record GraceExpired(Address Peer);
|
||||
|
||||
/// <summary>Timer: the direct-shutdown fallback after a restart was requested.</summary>
|
||||
internal sealed class FallbackShutdown
|
||||
{
|
||||
/// <summary>The singleton instance.</summary>
|
||||
public static readonly FallbackShutdown Instance = new();
|
||||
|
||||
private FallbackShutdown()
|
||||
{
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A peer identity independent of the transport scheme (the throttler adapter rewrites it).</summary>
|
||||
private readonly record struct PeerKey(string System, string? Host, int? Port)
|
||||
{
|
||||
public static PeerKey Of(Address address) => new(address.System, address.Host, address.Port);
|
||||
}
|
||||
|
||||
private sealed class ProbeState
|
||||
{
|
||||
public required Address Address { get; init; }
|
||||
public TimeSpan NextInterval { get; set; }
|
||||
public ICancelable? Timer { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Phase of a detected split with one peer.</summary>
|
||||
private enum SplitPhase
|
||||
{
|
||||
/// <summary>S2 seen: the peer's old incarnation is alive; it gets a grace to restart itself.</summary>
|
||||
Suspect,
|
||||
|
||||
/// <summary>Tiebreak said stay: the peer is expected to restart; it gets a grace to do so.</summary>
|
||||
Winner,
|
||||
}
|
||||
|
||||
private sealed class SplitState
|
||||
{
|
||||
public required SplitPhase Phase { get; set; }
|
||||
public required ICancelable Timer { get; set; }
|
||||
public DateTime LastEvidenceUtc { get; set; }
|
||||
}
|
||||
|
||||
private readonly ShutOutRestartSettings _settings;
|
||||
private readonly Dictionary<PeerKey, Member> _members = new();
|
||||
private readonly HashSet<PeerKey> _knownPeers = new();
|
||||
private readonly HashSet<PeerKey> _ignoredLogged = new();
|
||||
private readonly Dictionary<PeerKey, ProbeState> _probing = new();
|
||||
private readonly Dictionary<PeerKey, SplitState> _splits = new();
|
||||
private bool _restarting;
|
||||
|
||||
/// <summary>Creates the base logic.</summary>
|
||||
/// <param name="settings">The settings.</param>
|
||||
protected ShutOutRestartBase(ShutOutRestartSettings settings)
|
||||
{
|
||||
_settings = settings;
|
||||
}
|
||||
|
||||
/// <summary>Decision log.</summary>
|
||||
protected ILoggingAdapter Log { get; } = Context.GetLogger();
|
||||
|
||||
/// <summary>This node's address.</summary>
|
||||
public abstract Address SelfAddress { get; }
|
||||
|
||||
/// <summary>The scheduler used for the probe and grace timers.</summary>
|
||||
public abstract IScheduler Scheduler { get; }
|
||||
|
||||
/// <summary>The configured seed nodes; peers of this pair even before they were ever seen as members.</summary>
|
||||
protected abstract IEnumerable<Address> SeedNodes { get; }
|
||||
|
||||
/// <summary>The clock; overridable for tests.</summary>
|
||||
protected virtual DateTime UtcNow => DateTime.UtcNow;
|
||||
|
||||
/// <summary>Restarts this node (called at most once).</summary>
|
||||
protected abstract void RestartSelf();
|
||||
|
||||
/// <summary>Sends one probe to <paramref name="peer"/>.</summary>
|
||||
/// <param name="peer">The removed peer.</param>
|
||||
protected abstract void Probe(Address peer);
|
||||
|
||||
/// <summary>Runs the direct-shutdown fallback, if the restart has not taken effect.</summary>
|
||||
protected virtual void OnFallbackShutdown()
|
||||
{
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Whether an association failure is S2: the handshake met a UID this node has quarantined.
|
||||
/// Akka.Remote 1.5.62 carries no typed reason outward — the protocol layer's
|
||||
/// <c>ForbiddenUidReason</c> becomes "The remote system has a UID that has been quarantined.
|
||||
/// Association aborted." and the endpoint layer's <c>HopelessAssociation</c> says "The remote
|
||||
/// system [..] has a UID [..] that has been quarantined." — so the cause chain is matched on that
|
||||
/// wording. Pinned by the in-process scenario tests, which fail if an Akka upgrade rewords it.
|
||||
/// </summary>
|
||||
/// <param name="cause">The association error's cause.</param>
|
||||
/// <returns><see langword="true"/> when the failure is a refused, quarantined UID.</returns>
|
||||
internal static bool IsQuarantinedUidRefusal(Exception? cause)
|
||||
{
|
||||
for (var depth = 0; cause is not null && depth < 8; depth++)
|
||||
{
|
||||
var message = cause.Message;
|
||||
if (message.Contains("has a UID", StringComparison.Ordinal)
|
||||
&& message.Contains("that has been quarantined", StringComparison.Ordinal))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (cause is AggregateException aggregate && aggregate.InnerExceptions.Any(IsQuarantinedUidRefusal))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
cause = cause.InnerException;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
foreach (var seed in SeedNodes)
|
||||
{
|
||||
AddKnownPeer(seed);
|
||||
}
|
||||
|
||||
Log.Info(
|
||||
"[leader-auto-down] shut-out restart is ON for {0}: a node split from its pair peer restarts itself — at once if only it "
|
||||
+ "was removed, otherwise the higher address restarts after a {1} grace (ScadaBridge#132)",
|
||||
SelfAddress, _settings.Grace);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
CancelAllTimers();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void OnReceive(object message)
|
||||
{
|
||||
switch (message)
|
||||
{
|
||||
case ClusterEvent.CurrentClusterState state:
|
||||
_members.Clear();
|
||||
foreach (var member in state.Members)
|
||||
{
|
||||
TrackMember(member);
|
||||
}
|
||||
break;
|
||||
|
||||
case ClusterEvent.MemberRemoved removed:
|
||||
_members.Remove(PeerKey.Of(removed.Member.Address));
|
||||
if (removed.PreviousStatus == MemberStatus.Down && !IsSelf(removed.Member.Address))
|
||||
{
|
||||
StartProbing(removed.Member.Address);
|
||||
}
|
||||
break;
|
||||
|
||||
case ClusterEvent.IMemberEvent memberEvent:
|
||||
TrackMember(memberEvent.Member);
|
||||
break;
|
||||
|
||||
case ThisActorSystemQuarantinedEvent shutOut:
|
||||
OnShutOutByPeer(shutOut.RemoteAddress);
|
||||
break;
|
||||
|
||||
case AssociationErrorEvent error:
|
||||
if (IsQuarantinedUidRefusal(error.Cause))
|
||||
{
|
||||
OnPeerStillRunningRemovedIncarnation(error.RemoteAddress);
|
||||
}
|
||||
break;
|
||||
|
||||
case ProbeTick tick:
|
||||
OnProbeTick(tick.Peer);
|
||||
break;
|
||||
|
||||
case GraceExpired expired:
|
||||
OnGraceExpired(expired.Peer);
|
||||
break;
|
||||
|
||||
case FallbackShutdown:
|
||||
OnFallbackShutdown();
|
||||
break;
|
||||
|
||||
case ActorIdentity:
|
||||
// A probe answered by a reachable incarnation — nothing to do.
|
||||
break;
|
||||
|
||||
default:
|
||||
Unhandled(message);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool IsLiveStatus(MemberStatus status) =>
|
||||
status is MemberStatus.Joining or MemberStatus.WeaklyUp or MemberStatus.Up or MemberStatus.Leaving;
|
||||
|
||||
private bool IsSelf(Address address) => PeerKey.Of(address) == PeerKey.Of(SelfAddress);
|
||||
|
||||
private void AddKnownPeer(Address address)
|
||||
{
|
||||
if (!IsSelf(address))
|
||||
{
|
||||
_knownPeers.Add(PeerKey.Of(address));
|
||||
}
|
||||
}
|
||||
|
||||
private void TrackMember(Member member)
|
||||
{
|
||||
var key = PeerKey.Of(member.Address);
|
||||
_members[key] = member;
|
||||
AddKnownPeer(member.Address);
|
||||
if (IsLiveStatus(member.Status) && _probing.Remove(key, out var probe))
|
||||
{
|
||||
probe.Timer?.Cancel();
|
||||
}
|
||||
}
|
||||
|
||||
private bool IsLiveMember(Address address, out MemberStatus status)
|
||||
{
|
||||
if (_members.TryGetValue(PeerKey.Of(address), out var member) && IsLiveStatus(member.Status))
|
||||
{
|
||||
status = member.Status;
|
||||
return true;
|
||||
}
|
||||
|
||||
status = MemberStatus.Removed;
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>Common gate for both signals: ignore after a restart, for self, and for strangers.</summary>
|
||||
private bool IsRelevantPeer(Address peer, string what)
|
||||
{
|
||||
if (_restarting || IsSelf(peer))
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
var key = PeerKey.Of(peer);
|
||||
if (_knownPeers.Contains(key))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
if (_ignoredLogged.Add(key))
|
||||
{
|
||||
Log.Info("[leader-auto-down] {0} {1}, but it is not a seed or member of this cluster; ignored", peer, what);
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>S1: <paramref name="peer"/> has quarantined this node.</summary>
|
||||
private void OnShutOutByPeer(Address peer)
|
||||
{
|
||||
if (!IsRelevantPeer(peer, $"has quarantined this node ({SelfAddress})"))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (IsLiveMember(peer, out var status))
|
||||
{
|
||||
Log.Warning(
|
||||
"[leader-auto-down] shut out by {0}: it has removed and quarantined this node ({1}) while this node still counts it as a member ({2}), "
|
||||
+ "so only this side is shut out; tiebreak not needed — this node RESTARTS to rejoin as a fresh incarnation",
|
||||
peer, SelfAddress, status);
|
||||
Restart();
|
||||
return;
|
||||
}
|
||||
|
||||
if (_splits.TryGetValue(PeerKey.Of(peer), out var split))
|
||||
{
|
||||
split.LastEvidenceUtc = UtcNow;
|
||||
if (split.Phase == SplitPhase.Winner)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
// Suspect → the peer has quarantined us too: both sides are shut out, decide now.
|
||||
split.Timer.Cancel();
|
||||
_splits.Remove(PeerKey.Of(peer));
|
||||
}
|
||||
|
||||
Tiebreak(peer, $"shut out by {peer}: it has quarantined this node ({SelfAddress}), and this node has removed it too");
|
||||
}
|
||||
|
||||
/// <summary>S2: <paramref name="peer"/> is still running the incarnation this node removed.</summary>
|
||||
private void OnPeerStillRunningRemovedIncarnation(Address peer)
|
||||
{
|
||||
if (!IsRelevantPeer(peer, "refused a handshake with a UID this node has quarantined"))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
var key = PeerKey.Of(peer);
|
||||
if (_splits.TryGetValue(key, out var split))
|
||||
{
|
||||
split.LastEvidenceUtc = UtcNow;
|
||||
return;
|
||||
}
|
||||
|
||||
if (IsLiveMember(peer, out _))
|
||||
{
|
||||
// A live member with a quarantined UID is a transient of Akka's own quarantine path
|
||||
// (it is about to be marked terminated and downed); the downing provider handles it.
|
||||
return;
|
||||
}
|
||||
|
||||
Log.Warning(
|
||||
"[leader-auto-down] split from {0}: it is still running the incarnation this node ({1}) removed, so this node has shut it out. "
|
||||
+ "Giving it {2} to restart itself before the tiebreak (a frozen node that was removed restarts on its own)",
|
||||
peer, SelfAddress, _settings.Grace);
|
||||
StartSplit(peer, SplitPhase.Suspect);
|
||||
}
|
||||
|
||||
private void Tiebreak(Address peer, string why)
|
||||
{
|
||||
if (Member.AddressOrdering.Compare(SelfAddress, peer) > 0)
|
||||
{
|
||||
Log.Warning(
|
||||
"[leader-auto-down] {0}; both sides are shut out. Tiebreak says RESTART: this node ({1}) has the higher address",
|
||||
why, SelfAddress);
|
||||
Restart();
|
||||
return;
|
||||
}
|
||||
|
||||
Log.Warning(
|
||||
"[leader-auto-down] {0}; both sides are shut out. Tiebreak says STAY: this node ({1}) has the lower address, so {2} restarts. "
|
||||
+ "This node restarts too only if {2} has not rejoined and is still split from it after {3}",
|
||||
why, SelfAddress, peer, _settings.Grace);
|
||||
StartSplit(peer, SplitPhase.Winner);
|
||||
}
|
||||
|
||||
private void StartSplit(Address peer, SplitPhase phase)
|
||||
{
|
||||
_splits[PeerKey.Of(peer)] = new SplitState
|
||||
{
|
||||
Phase = phase,
|
||||
Timer = Scheduler.ScheduleTellOnceCancelable(_settings.Grace, Self, new GraceExpired(peer), Self),
|
||||
LastEvidenceUtc = UtcNow,
|
||||
};
|
||||
|
||||
// Probe at the short interval during the grace, so "still split" is measured.
|
||||
StartProbing(peer, restartBackOff: true);
|
||||
}
|
||||
|
||||
private void OnGraceExpired(Address peer)
|
||||
{
|
||||
if (_restarting || !_splits.Remove(PeerKey.Of(peer), out var split))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
if (IsLiveMember(peer, out var status))
|
||||
{
|
||||
Log.Info(
|
||||
"[leader-auto-down] {0} rejoined ({1}) within the shut-out grace; this node ({2}) stays",
|
||||
peer, status, SelfAddress);
|
||||
return;
|
||||
}
|
||||
|
||||
var sinceLast = UtcNow - split.LastEvidenceUtc;
|
||||
if (sinceLast > _settings.RecentWindow)
|
||||
{
|
||||
Log.Info(
|
||||
"[leader-auto-down] {0} has not rejoined yet, but nothing of its old incarnation has been seen for {1:0}s (it is restarting or down); "
|
||||
+ "this node ({2}) stays",
|
||||
peer, sinceLast.TotalSeconds, SelfAddress);
|
||||
return;
|
||||
}
|
||||
|
||||
if (split.Phase == SplitPhase.Suspect)
|
||||
{
|
||||
Tiebreak(peer,
|
||||
$"split from {peer}: its old incarnation is still running {_settings.Grace} later and it has not restarted itself");
|
||||
return;
|
||||
}
|
||||
|
||||
Log.Warning(
|
||||
"[leader-auto-down] {0} is still split from this node ({1}) {2} after the tiebreak (last seen {3:0}s ago) and has not rejoined, "
|
||||
+ "so this node is the stale side after all; this node RESTARTS",
|
||||
peer, SelfAddress, _settings.Grace, sinceLast.TotalSeconds);
|
||||
Restart();
|
||||
}
|
||||
|
||||
private void StartProbing(Address peer, bool restartBackOff = false)
|
||||
{
|
||||
var key = PeerKey.Of(peer);
|
||||
if (_probing.TryGetValue(key, out var existing))
|
||||
{
|
||||
if (!restartBackOff)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
existing.Timer?.Cancel();
|
||||
}
|
||||
|
||||
var state = existing ?? new ProbeState { Address = peer };
|
||||
state.NextInterval = _settings.ProbeInterval;
|
||||
state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self);
|
||||
_probing[key] = state;
|
||||
}
|
||||
|
||||
private void OnProbeTick(Address peer)
|
||||
{
|
||||
var key = PeerKey.Of(peer);
|
||||
if (_restarting || !_probing.TryGetValue(key, out var state))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
Probe(state.Address);
|
||||
|
||||
// Fixed short interval while a grace runs (to measure "still split"); otherwise back off.
|
||||
state.NextInterval = _splits.ContainsKey(key)
|
||||
? _settings.ProbeInterval
|
||||
: TimeSpan.FromTicks(Math.Min(state.NextInterval.Ticks * 2, _settings.ProbeMaxInterval.Ticks));
|
||||
state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self);
|
||||
}
|
||||
|
||||
private void Restart()
|
||||
{
|
||||
_restarting = true;
|
||||
CancelAllTimers();
|
||||
_probing.Clear();
|
||||
_splits.Clear();
|
||||
RestartSelf();
|
||||
}
|
||||
|
||||
private void CancelAllTimers()
|
||||
{
|
||||
foreach (var probe in _probing.Values)
|
||||
{
|
||||
probe.Timer?.Cancel();
|
||||
}
|
||||
|
||||
foreach (var split in _splits.Values)
|
||||
{
|
||||
split.Timer.Cancel();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -157,6 +157,11 @@ Log.Logger = ZB.MOM.WW.ScadaBridge.Host.ScadaBridgeLogging
|
||||
// The catch below uses it to tell that routine stop from a genuine fatal exit (ScadaBridge#76).
|
||||
var hostStopping = CancellationToken.None;
|
||||
|
||||
// The Akka hosted service, captured with hostStopping (RunAsync disposes the host's service
|
||||
// provider), so the finally block can turn an Akka watchdog stop (self-down / ScadaBridge#132
|
||||
// shut-out restart) into a non-zero exit code whichever way RunAsync unwound.
|
||||
AkkaHostedService? akkaHostedService = null;
|
||||
|
||||
try
|
||||
{
|
||||
Log.Information("Starting ScadaBridge host as {Role} on {Hostname}", nodeRole, nodeHostname);
|
||||
@@ -670,6 +675,7 @@ try
|
||||
|
||||
var app = builder.Build();
|
||||
hostStopping = app.Lifetime.ApplicationStopping;
|
||||
akkaHostedService = app.Services.GetService<AkkaHostedService>();
|
||||
|
||||
// Apply or validate database migrations (skip when running in test harness)
|
||||
if (!string.Equals(configuration["ScadaBridge:Database:SkipMigrations"], "true", StringComparison.OrdinalIgnoreCase))
|
||||
@@ -864,6 +870,7 @@ try
|
||||
|
||||
var app = builder.Build();
|
||||
hostStopping = app.Lifetime.ApplicationStopping;
|
||||
akkaHostedService = app.Services.GetService<AkkaHostedService>();
|
||||
|
||||
// Endpoint routing middleware. The gRPC service mapping below and the
|
||||
// /metrics scrape endpoint both run on endpoint routing, so UseRouting()
|
||||
@@ -934,6 +941,8 @@ catch (Exception ex)
|
||||
}
|
||||
finally
|
||||
{
|
||||
Program.SetSelfDownExitCode(akkaHostedService);
|
||||
|
||||
await Log.CloseAndFlushAsync();
|
||||
}
|
||||
|
||||
@@ -955,6 +964,25 @@ public partial class Program
|
||||
internal static bool IsGracefulShutdown(Exception ex, CancellationToken hostStopping) =>
|
||||
ex is OperationCanceledException && hostStopping.IsCancellationRequested;
|
||||
|
||||
/// <summary>
|
||||
/// After <c>RunAsync</c> returns: when the stop was the Akka watchdog's (the ActorSystem
|
||||
/// terminated outside host shutdown — a self-down or a ScadaBridge#132 shut-out restart), exit
|
||||
/// with the non-zero <see cref="AkkaHostedService.SelfDownExitCode"/> so the service supervisor
|
||||
/// restarts the node. Docker <c>restart: unless-stopped</c> restarts on any exit; Windows service
|
||||
/// recovery gets the code through the service lifetime (<c>AkkaHostedService.MarkServiceSelfDownExit</c>).
|
||||
/// </summary>
|
||||
/// <param name="akka">The host's Akka hosted service; null if the host never built.</param>
|
||||
internal static void SetSelfDownExitCode(AkkaHostedService? akka)
|
||||
{
|
||||
if (akka is { TerminatedOutsideHostShutdown: true })
|
||||
{
|
||||
Environment.ExitCode = AkkaHostedService.SelfDownExitCode;
|
||||
Log.Warning(
|
||||
"ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code {ExitCode} so the service supervisor restarts it",
|
||||
AkkaHostedService.SelfDownExitCode);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the
|
||||
/// <c>ASPNETCORE_URLS</c> / <c>--urls</c> value). Port-only view of
|
||||
|
||||
+28
@@ -327,4 +327,32 @@ public class ClusterOptionsValidatorTests
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("'leader-auto-down'", result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132: a non-positive shut-out grace is refused under leader-auto-down with the
|
||||
/// restart on (it would make both sides of a healed partition restart together), and ignored
|
||||
/// when the restart is off or another strategy is selected.
|
||||
/// </summary>
|
||||
/// <param name="strategy">The strategy.</param>
|
||||
/// <param name="restart">The shut-out restart switch.</param>
|
||||
/// <param name="expectValid">Whether validation must pass.</param>
|
||||
[Theory]
|
||||
[InlineData("leader-auto-down", true, false)]
|
||||
[InlineData("leader-auto-down", false, true)]
|
||||
[InlineData("auto-down", true, true)]
|
||||
public void LeaderAutoDown_ZeroShutOutGrace(string strategy, bool restart, bool expectValid)
|
||||
{
|
||||
var options = ValidOptions();
|
||||
options.SplitBrainResolverStrategy = strategy;
|
||||
options.LeaderAutoDown = new LeaderAutoDownOptions { RestartWhenShutOut = restart, ShutOutGrace = TimeSpan.Zero };
|
||||
|
||||
var result = new ClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
Assert.Equal(expectValid, result.Succeeded);
|
||||
if (!expectValid)
|
||||
{
|
||||
Assert.Contains("ShutOutGrace", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -322,4 +322,46 @@ public class HoconBuilderTests
|
||||
Assert.DoesNotContain("leader-auto-down", hocon);
|
||||
Assert.DoesNotContain(nameof(LeaderAutoDowningProvider), hocon);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132: leader-auto-down carries the shut-out restart switch (default ON) and its
|
||||
/// grace into the provider's HOCON block, and the provider reads back exactly those values.
|
||||
/// </summary>
|
||||
/// <param name="enabled">The configured switch.</param>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void BuildHocon_LeaderAutoDown_EmitsShutOutRestartSettings(bool enabled)
|
||||
{
|
||||
var cluster = DefaultCluster();
|
||||
cluster.SplitBrainResolverStrategy = "leader-auto-down";
|
||||
cluster.LeaderAutoDown = new LeaderAutoDownOptions
|
||||
{
|
||||
RestartWhenShutOut = enabled,
|
||||
ShutOutGrace = TimeSpan.FromMilliseconds(45500),
|
||||
};
|
||||
|
||||
var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
|
||||
DefaultNode(), cluster, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
|
||||
|
||||
var settings = ShutOutRestartSettings.FromConfig(config);
|
||||
Assert.Equal(enabled, settings.Enabled);
|
||||
Assert.Equal(TimeSpan.FromMilliseconds(45500), settings.Grace);
|
||||
}
|
||||
|
||||
/// <summary>ScadaBridge#132: the shut-out restart is ON by default when leader-auto-down is selected.</summary>
|
||||
[Fact]
|
||||
public void BuildHocon_LeaderAutoDown_ShutOutRestartOnByDefault()
|
||||
{
|
||||
var cluster = DefaultCluster();
|
||||
cluster.SplitBrainResolverStrategy = "leader-auto-down";
|
||||
|
||||
var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
|
||||
DefaultNode(), cluster, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
|
||||
|
||||
Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
|
||||
Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,447 @@
|
||||
using System.Collections.Immutable;
|
||||
using System.Reflection;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Akka.Remote;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Akka.Util;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132: the decision logic of <see cref="ShutOutRestartBase"/>. Cluster events and
|
||||
/// <see cref="ThisActorSystemQuarantinedEvent"/> are fed straight to the actor; the restart and
|
||||
/// probe seams report to the test actor. Covers the single-side shut-out (restart, no tiebreak),
|
||||
/// the tiebreak in both orders, the both-sides case where exactly one of two nodes restarts, the
|
||||
/// winner's grace, and the probing that makes a healed partition produce the event at all. The
|
||||
/// real two-node behaviour is proven in IntegrationTests (<c>ShutOutRestartScenarioTests</c>).
|
||||
/// </summary>
|
||||
public class ShutOutRestartTests : TestKit
|
||||
{
|
||||
/// <summary>INFO so the decision-log assertions see every line.</summary>
|
||||
public ShutOutRestartTests()
|
||||
: base("akka.loglevel = INFO")
|
||||
{
|
||||
}
|
||||
|
||||
// "a" < "b" < "c" by Akka's member address ordering (host, then port).
|
||||
private static readonly Address NodeA = new("akka.tcp", "scadabridge", "a", 8081);
|
||||
private static readonly Address NodeB = new("akka.tcp", "scadabridge", "b", 8081);
|
||||
private static readonly Address Stranger = new("akka.tcp", "scadabridge", "z", 9999);
|
||||
|
||||
private static readonly ShutOutRestartSettings Fast = new(
|
||||
Enabled: true,
|
||||
Grace: TimeSpan.FromSeconds(1.5),
|
||||
ProbeInterval: TimeSpan.FromMilliseconds(200),
|
||||
ProbeMaxInterval: TimeSpan.FromMilliseconds(800));
|
||||
|
||||
private sealed record Restarted(Address Self);
|
||||
|
||||
private sealed record Probed(Address Self, Address Peer);
|
||||
|
||||
/// <summary>Test double: restart and probe report to a probe actor.</summary>
|
||||
private sealed class TestShutOutRestart : ShutOutRestartBase
|
||||
{
|
||||
private readonly IActorRef _probe;
|
||||
private readonly Address[] _seeds;
|
||||
|
||||
public TestShutOutRestart(Address self, Address[] seeds, ShutOutRestartSettings settings, IActorRef probe)
|
||||
: base(settings)
|
||||
{
|
||||
SelfAddress = self;
|
||||
_seeds = seeds;
|
||||
_probe = probe;
|
||||
}
|
||||
|
||||
public override Address SelfAddress { get; }
|
||||
|
||||
public override IScheduler Scheduler => Context.System.Scheduler;
|
||||
|
||||
protected override IEnumerable<Address> SeedNodes => _seeds;
|
||||
|
||||
protected override void RestartSelf() => _probe.Tell(new Restarted(SelfAddress));
|
||||
|
||||
protected override void Probe(Address peer) => _probe.Tell(new Probed(SelfAddress, peer));
|
||||
}
|
||||
|
||||
private static Member CreateMember(Address address, MemberStatus status)
|
||||
{
|
||||
var create = typeof(Member).GetMethod(
|
||||
"Create",
|
||||
BindingFlags.Static | BindingFlags.NonPublic,
|
||||
new[] { typeof(UniqueAddress), typeof(int), typeof(MemberStatus), typeof(ImmutableHashSet<string>), typeof(AppVersion) })
|
||||
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(…) not found — Akka internals changed.");
|
||||
return (Member)create.Invoke(null, new object[]
|
||||
{
|
||||
new UniqueAddress(address, 1), 1, status, ImmutableHashSet<string>.Empty, AppVersion.Zero,
|
||||
})!;
|
||||
}
|
||||
|
||||
private static ClusterEvent.CurrentClusterState StateWith(params Member[] members) =>
|
||||
new(
|
||||
ImmutableSortedSet.CreateRange(members),
|
||||
ImmutableHashSet<Member>.Empty,
|
||||
ImmutableHashSet<Address>.Empty,
|
||||
null,
|
||||
ImmutableDictionary<string, Address>.Empty);
|
||||
|
||||
private IActorRef Node(Address self, Address peer, IActorRef? probe = null, ShutOutRestartSettings? settings = null) =>
|
||||
Sys.ActorOf(Props.Create(() => new TestShutOutRestart(
|
||||
self, new[] { self, peer }, settings ?? Fast, probe ?? TestActor)));
|
||||
|
||||
/// <summary>Both nodes see each other Up, then each removes the other after downing it (the partition outcome).</summary>
|
||||
private static void BothRemovedEachOther(IActorRef node, Address self, Address peer)
|
||||
{
|
||||
node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up)));
|
||||
node.Tell(new ClusterEvent.MemberRemoved(CreateMember(peer, MemberStatus.Removed), MemberStatus.Down));
|
||||
}
|
||||
|
||||
private static ThisActorSystemQuarantinedEvent ShutOutBy(Address peer, Address self) => new(self, peer);
|
||||
|
||||
/// <summary>S2 as Akka.Remote 1.5.62 surfaces it: the endpoint writer's InvalidAssociation wrapping the protocol refusal.</summary>
|
||||
private static AssociationErrorEvent StillRunningRemoved(Address peer, Address self) =>
|
||||
new(
|
||||
new Exception(
|
||||
$"Association failed with {peer}",
|
||||
new AggregateException(new Exception("The remote system has a UID that has been quarantined. Association aborted."))),
|
||||
self, peer, inbound: false, Akka.Event.LogLevel.WarningLevel);
|
||||
|
||||
/// <summary>Everything <paramref name="probe"/> receives during <paramref name="window"/>.</summary>
|
||||
private static List<object> Drain(Akka.TestKit.TestProbe probe, TimeSpan window)
|
||||
{
|
||||
var received = new List<object>();
|
||||
var until = DateTime.UtcNow + window;
|
||||
while (true)
|
||||
{
|
||||
var left = until - DateTime.UtcNow;
|
||||
if (left <= TimeSpan.Zero)
|
||||
{
|
||||
return received;
|
||||
}
|
||||
|
||||
if (probe.TryReceiveOne(out var envelope, left))
|
||||
{
|
||||
received.Add(envelope.Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Single side (the frozen node): the peer is still a live member in this node's view, so only
|
||||
/// this side can be shut out — it restarts at once, whichever address is higher.
|
||||
/// </summary>
|
||||
/// <param name="selfIsA">Whether this node is the lower-address node.</param>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void ShutOut_PeerStillAMember_RestartsWithoutTiebreak(bool selfIsA)
|
||||
{
|
||||
var (self, peer) = selfIsA ? (NodeA, NodeB) : (NodeB, NodeA);
|
||||
var node = Node(self, peer);
|
||||
node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up)));
|
||||
|
||||
EventFilter.Warning(contains: "tiebreak not needed").ExpectOne(() => node.Tell(ShutOutBy(peer, self)));
|
||||
ExpectMsg(new Restarted(self));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||
}
|
||||
|
||||
/// <summary>Tiebreak, higher address: both sides removed each other; the higher address restarts at once.</summary>
|
||||
[Fact]
|
||||
public void ShutOut_PeerRemoved_HigherAddress_Restarts()
|
||||
{
|
||||
var node = Node(NodeB, NodeA);
|
||||
BothRemovedEachOther(node, NodeB, NodeA);
|
||||
|
||||
EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB)));
|
||||
ExpectMsg(new Restarted(NodeB));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Tiebreak, lower address: it stays; it keeps probing during the grace, and at the end of the
|
||||
/// grace — the peer silent, i.e. it restarted — it still stays.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void ShutOut_PeerRemoved_LowerAddress_Stays_AndStaysAfterGraceWhenPeerWentQuiet()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeA, NodeB, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeA, NodeB);
|
||||
|
||||
EventFilter.Warning(contains: "Tiebreak says STAY").ExpectOne(() => node.Tell(ShutOutBy(NodeB, NodeA)));
|
||||
EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { });
|
||||
|
||||
var received = Drain(probe, TimeSpan.FromMilliseconds(500));
|
||||
Assert.Contains(new Probed(NodeA, NodeB), received); // it kept probing its peer
|
||||
Assert.DoesNotContain(new Restarted(NodeA), received); // and never restarted
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Tiebreak, lower address, but the peer keeps refusing for the whole grace and never rejoins:
|
||||
/// the lower node was the stale side after all (single-sided with the lower address) and restarts.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task ShutOut_PeerRemoved_LowerAddress_StillRefusedAfterGrace_Restarts()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeA, NodeB, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeA, NodeB);
|
||||
node.Tell(ShutOutBy(NodeB, NodeA));
|
||||
|
||||
var until = DateTime.UtcNow + Fast.Grace + TimeSpan.FromSeconds(1);
|
||||
var restarted = false;
|
||||
while (DateTime.UtcNow < until && !restarted)
|
||||
{
|
||||
node.Tell(ShutOutBy(NodeB, NodeA)); // every probe is refused
|
||||
await Task.Delay(100);
|
||||
while (probe.HasMessages)
|
||||
{
|
||||
restarted |= probe.ReceiveOne(TimeSpan.Zero) is Restarted;
|
||||
}
|
||||
}
|
||||
|
||||
Assert.True(restarted, "the lower node never restarted although its peer kept refusing it past the grace");
|
||||
}
|
||||
|
||||
/// <summary>Tiebreak, lower address: the peer rejoins during the grace; the lower node stays.</summary>
|
||||
[Fact]
|
||||
public void ShutOut_PeerRemoved_LowerAddress_PeerRejoins_Stays()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeA, NodeB, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeA, NodeB);
|
||||
node.Tell(ShutOutBy(NodeB, NodeA));
|
||||
node.Tell(ShutOutBy(NodeB, NodeA)); // refused right up to the restart of the peer
|
||||
node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeB, MemberStatus.Up)));
|
||||
|
||||
EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { });
|
||||
Assert.DoesNotContain(new Restarted(NodeA), Drain(probe, TimeSpan.FromMilliseconds(500)));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Both sides shut out (a healed partition): two nodes, each having removed the other, both
|
||||
/// receive the event — exactly ONE restarts, and it is the higher address. Run in both
|
||||
/// delivery orders so the outcome cannot depend on who hears first.
|
||||
/// </summary>
|
||||
/// <param name="higherHearsFirst">Whether the higher-address node receives its event first.</param>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public void BothSidesShutOut_ExactlyOneRestarts_TheHigherAddress(bool higherHearsFirst)
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var a = Node(NodeA, NodeB, probe.Ref);
|
||||
var b = Node(NodeB, NodeA, probe.Ref);
|
||||
BothRemovedEachOther(a, NodeA, NodeB);
|
||||
BothRemovedEachOther(b, NodeB, NodeA);
|
||||
|
||||
if (higherHearsFirst)
|
||||
{
|
||||
b.Tell(ShutOutBy(NodeA, NodeB));
|
||||
a.Tell(ShutOutBy(NodeB, NodeA));
|
||||
}
|
||||
else
|
||||
{
|
||||
a.Tell(ShutOutBy(NodeB, NodeA));
|
||||
b.Tell(ShutOutBy(NodeA, NodeB));
|
||||
}
|
||||
|
||||
// Within the grace, the only restart is B's.
|
||||
var restarts = Drain(probe, Fast.Grace - TimeSpan.FromMilliseconds(300)).OfType<Restarted>().ToList();
|
||||
Assert.Equal(new[] { new Restarted(NodeB) }, restarts);
|
||||
}
|
||||
|
||||
/// <summary>The same event delivered many times restarts once.</summary>
|
||||
[Fact]
|
||||
public void RepeatedShutOut_RestartsOnce()
|
||||
{
|
||||
var node = Node(NodeB, NodeA);
|
||||
node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up)));
|
||||
for (var i = 0; i < 5; i++)
|
||||
{
|
||||
node.Tell(ShutOutBy(NodeA, NodeB));
|
||||
}
|
||||
|
||||
ExpectMsg(new Restarted(NodeB));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>A quarantine by an address that is neither a seed nor a member is not this pair's business.</summary>
|
||||
[Fact]
|
||||
public void ShutOutByStranger_IsIgnored()
|
||||
{
|
||||
var node = Node(NodeB, NodeA);
|
||||
node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up)));
|
||||
|
||||
EventFilter.Info(contains: "not a seed or member of this cluster").ExpectOne(() => node.Tell(ShutOutBy(Stranger, NodeB)));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A peer removed after being DOWNED is probed with a back-off until it is a member again; a
|
||||
/// peer that LEFT (Exiting → Removed) is not probed.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void DownedPeer_IsProbedUntilItRejoins_LeftPeerIsNot()
|
||||
{
|
||||
var node = Node(NodeA, NodeB);
|
||||
node.Tell(StateWith(CreateMember(NodeA, MemberStatus.Up), CreateMember(NodeB, MemberStatus.Up)));
|
||||
|
||||
node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Exiting));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(600));
|
||||
|
||||
node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Down));
|
||||
ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1));
|
||||
ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1));
|
||||
|
||||
node.Tell(new ClusterEvent.MemberJoined(CreateMember(NodeB, MemberStatus.Joining)));
|
||||
ReceiveWhile<object>(TimeSpan.FromMilliseconds(100), m => m); // a tick already in flight
|
||||
ExpectNoMsg(TimeSpan.FromSeconds(1.5));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// S2 on the higher node (it removed a peer whose old incarnation is still alive): no restart
|
||||
/// during the grace — a frozen peer gets the chance to restart itself — then, still split, the
|
||||
/// tiebreak restarts it.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task StillRunningRemoved_HigherAddress_WaitsGrace_ThenRestartsIfStillSplit()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeB, NodeA, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeB, NodeA);
|
||||
|
||||
EventFilter.Warning(contains: "Giving it").ExpectOne(() => node.Tell(StillRunningRemoved(NodeA, NodeB)));
|
||||
var started = DateTime.UtcNow;
|
||||
var restartedAt = (DateTime?)null;
|
||||
while (DateTime.UtcNow - started < Fast.Grace + TimeSpan.FromSeconds(1.5) && restartedAt is null)
|
||||
{
|
||||
node.Tell(StillRunningRemoved(NodeA, NodeB)); // every probe still meets the old incarnation
|
||||
await Task.Delay(100);
|
||||
while (probe.HasMessages)
|
||||
{
|
||||
if (probe.ReceiveOne(TimeSpan.Zero) is Restarted)
|
||||
{
|
||||
restartedAt = DateTime.UtcNow;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Assert.NotNull(restartedAt);
|
||||
Assert.True(restartedAt.Value - started >= Fast.Grace - TimeSpan.FromMilliseconds(200),
|
||||
$"restarted after {(restartedAt.Value - started).TotalSeconds:0.0}s, before the {Fast.Grace} grace");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The freeze survivor: S2 (the frozen node's old incarnation is alive), then the frozen node
|
||||
/// restarts itself and its old incarnation goes quiet — the survivor does NOT restart, even
|
||||
/// though it has the higher address.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void StillRunningRemoved_PeerGoesQuiet_NoRestartEvenWhenHigher()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeB, NodeA, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeB, NodeA);
|
||||
node.Tell(StillRunningRemoved(NodeA, NodeB));
|
||||
|
||||
EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { });
|
||||
Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500)));
|
||||
}
|
||||
|
||||
/// <summary>S2, then the peer's new incarnation joins during the grace: no restart.</summary>
|
||||
[Fact]
|
||||
public void StillRunningRemoved_PeerRejoins_NoRestart()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var node = Node(NodeB, NodeA, probe.Ref);
|
||||
BothRemovedEachOther(node, NodeB, NodeA);
|
||||
node.Tell(StillRunningRemoved(NodeA, NodeB));
|
||||
node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeA, MemberStatus.Up)));
|
||||
node.Tell(StillRunningRemoved(NodeA, NodeB)); // stale evidence from before the restart
|
||||
|
||||
EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { });
|
||||
Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500)));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A healed partition as Akka 1.5.62 really reports it — S2 on BOTH sides, no S1 — ends with
|
||||
/// exactly one restart, the higher address, after the grace; the lower stays.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task BothSidesStillRunningRemoved_ExactlyOneRestarts_TheHigherAddress()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var a = Node(NodeA, NodeB, probe.Ref);
|
||||
var b = Node(NodeB, NodeA, probe.Ref);
|
||||
BothRemovedEachOther(a, NodeA, NodeB);
|
||||
BothRemovedEachOther(b, NodeB, NodeA);
|
||||
|
||||
var restarts = new List<Restarted>();
|
||||
var until = DateTime.UtcNow + Fast.Grace + Fast.Grace - TimeSpan.FromMilliseconds(400);
|
||||
while (DateTime.UtcNow < until)
|
||||
{
|
||||
if (!restarts.Contains(new Restarted(NodeB)))
|
||||
{
|
||||
// Until B's old incarnation is gone, each side's probes meet the other's.
|
||||
a.Tell(StillRunningRemoved(NodeB, NodeA));
|
||||
b.Tell(StillRunningRemoved(NodeA, NodeB));
|
||||
}
|
||||
|
||||
await Task.Delay(100);
|
||||
while (probe.HasMessages)
|
||||
{
|
||||
if (probe.ReceiveOne(TimeSpan.Zero) is Restarted r)
|
||||
{
|
||||
restarts.Add(r);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Assert.Equal(new[] { new Restarted(NodeB) }, restarts);
|
||||
}
|
||||
|
||||
/// <summary>S1 arriving while an S2 grace runs: the peer has quarantined us too, so the tiebreak runs at once.</summary>
|
||||
[Fact]
|
||||
public void StillRunningRemoved_ThenShutOut_TiebreaksImmediately()
|
||||
{
|
||||
var node = Node(NodeB, NodeA);
|
||||
BothRemovedEachOther(node, NodeB, NodeA);
|
||||
node.Tell(StillRunningRemoved(NodeA, NodeB));
|
||||
|
||||
EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB)));
|
||||
FishForMessage(m => m is Restarted, TimeSpan.FromSeconds(1));
|
||||
}
|
||||
|
||||
/// <summary>S2 detection matches Akka 1.5.62's two wordings anywhere in the cause chain, and nothing else.</summary>
|
||||
[Fact]
|
||||
public void IsQuarantinedUidRefusal_MatchesAkkaWordingOnly()
|
||||
{
|
||||
Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal(
|
||||
new Exception("Association failed with x", new AggregateException(
|
||||
new Exception("The remote system has a UID that has been quarantined. Association aborted.")))));
|
||||
Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal(
|
||||
new Exception("The remote system [akka.tcp://scadabridge@b:8081] has a UID [42] that has been quarantined. Association aborted.")));
|
||||
Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(
|
||||
new Exception("Association failed with x", new Exception("Connection refused"))));
|
||||
Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(
|
||||
new Exception("The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.")));
|
||||
Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(null));
|
||||
}
|
||||
|
||||
/// <summary>Settings: defaults when keys are missing; the switch and durations are read when present.</summary>
|
||||
[Fact]
|
||||
public void Settings_FromConfig_DefaultsAndOverrides()
|
||||
{
|
||||
var defaults = ShutOutRestartSettings.FromConfig(ConfigurationFactory.Empty);
|
||||
Assert.Equal(new ShutOutRestartSettings(true, TimeSpan.FromSeconds(60), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60)), defaults);
|
||||
|
||||
var set = ShutOutRestartSettings.FromConfig(ConfigurationFactory.ParseString($@"
|
||||
{ShutOutRestartSettings.EnabledPath} = off
|
||||
{ShutOutRestartSettings.GracePath} = 2500ms
|
||||
{ShutOutRestartSettings.ProbeIntervalPath} = 1s
|
||||
{ShutOutRestartSettings.ProbeMaxIntervalPath} = 4s"));
|
||||
Assert.Equal(new ShutOutRestartSettings(false, TimeSpan.FromMilliseconds(2500), TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(4)), set);
|
||||
}
|
||||
}
|
||||
@@ -69,6 +69,8 @@ public sealed class UnexpectedTerminationTests
|
||||
await Task.Delay(500);
|
||||
|
||||
lifetime.Received(1).StopApplication();
|
||||
// ScadaBridge#132: Program turns this flag into the non-zero SelfDownExitCode.
|
||||
Assert.True(service.TerminatedOutsideHostShutdown);
|
||||
}
|
||||
|
||||
/// <summary>Verifies that a normal stop does not trigger an application stop.</summary>
|
||||
@@ -84,5 +86,19 @@ public sealed class UnexpectedTerminationTests
|
||||
await Task.Delay(500);
|
||||
|
||||
lifetime.DidNotReceive().StopApplication();
|
||||
Assert.False(service.TerminatedOutsideHostShutdown);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132: the self-down exit code is non-zero (Windows service recovery with
|
||||
/// failureflag 1 restarts only on a non-zero exit) and is only pushed into a Windows service
|
||||
/// lifetime — a console / docker lifetime is left alone.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void SelfDownExitCode_IsNonZero_AndOnlyAppliedToAServiceLifetime()
|
||||
{
|
||||
Assert.NotEqual(0, AkkaHostedService.SelfDownExitCode);
|
||||
Assert.False(AkkaHostedService.MarkServiceSelfDownExit(null));
|
||||
Assert.False(AkkaHostedService.MarkServiceSelfDownExit(Substitute.For<IHostLifetime>()));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Akka.Remote;
|
||||
using Akka.Remote.Transport;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#132, on real in-process two-node clusters built from the production
|
||||
/// <c>BuildHocon</c> (same harness as <see cref="LeaderAutoDownScenarioTests"/>, with Akka's
|
||||
/// throttler transport to cut and restore the network without killing either node):
|
||||
/// <list type="bullet">
|
||||
/// <item>the FROZEN-node shape — only one side downed the other before the network came back —
|
||||
/// ends with the shut-out node restarting (its <c>ActorSystem</c> terminates) and the other
|
||||
/// staying Up;</item>
|
||||
/// <item>a partition held past the window and then HEALED — each side downed the other — ends
|
||||
/// with exactly one restart, the higher address; a fresh incarnation on that address rejoins the
|
||||
/// survivor as a 2-member cluster, and the survivor is still Up after its grace;</item>
|
||||
/// <item>with the switch OFF, the healed partition keeps both sides up (the #116 gate-4 contract
|
||||
/// before #132) — the positive control that the restart comes from the feature.</item>
|
||||
/// </list>
|
||||
/// "Restart" in-process is the termination of the node's <c>ActorSystem</c>; the Host turns that
|
||||
/// into a process exit (covered by <c>UnexpectedTerminationTests</c>).
|
||||
/// </summary>
|
||||
public sealed class ShutOutRestartScenarioTests : IAsyncLifetime
|
||||
{
|
||||
private static readonly TimeSpan Heartbeat = TimeSpan.FromMilliseconds(500);
|
||||
private static readonly TimeSpan Threshold = TimeSpan.FromSeconds(2);
|
||||
|
||||
private readonly List<ActorSystem> _systems = new();
|
||||
|
||||
private ActorSystem StartNode(
|
||||
int selfPort, int peerPort, TimeSpan stableAfter, bool restartWhenShutOut = true, TimeSpan? grace = null)
|
||||
{
|
||||
const string scheme = "akka.trttl.tcp";
|
||||
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
|
||||
var clusterOptions = new ClusterOptions
|
||||
{
|
||||
SeedNodes = new List<string>
|
||||
{
|
||||
$"{scheme}://scadabridge@127.0.0.1:{selfPort}",
|
||||
$"{scheme}://scadabridge@127.0.0.1:{peerPort}",
|
||||
},
|
||||
SplitBrainResolverStrategy = "leader-auto-down",
|
||||
StableAfter = stableAfter,
|
||||
HeartbeatInterval = Heartbeat,
|
||||
FailureDetectionThreshold = Threshold,
|
||||
MinNrOfMembers = 1,
|
||||
LeaderAutoDown = new LeaderAutoDownOptions
|
||||
{
|
||||
RestartWhenShutOut = restartWhenShutOut,
|
||||
ShutOutGrace = grace ?? TimeSpan.FromSeconds(20),
|
||||
},
|
||||
};
|
||||
var hocon = AkkaHostedService.BuildHocon(
|
||||
nodeOptions, clusterOptions, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(60));
|
||||
var overrides = "akka.coordinated-shutdown.run-by-actor-system-terminate = off\n"
|
||||
+ "akka.remote.dot-netty.tcp.applied-adapters = [trttl]\n"
|
||||
+ "akka.remote.retry-gate-closed-for = 500ms\n"
|
||||
// Fast probes so a healed partition is noticed within the test's patience.
|
||||
+ "scadabridge.cluster.leader-auto-down.shut-out-probe-interval = 1s\n"
|
||||
+ "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval = 2s\n";
|
||||
var config = ConfigurationFactory.ParseString(overrides)
|
||||
.WithFallback(ConfigurationFactory.ParseString(hocon));
|
||||
var system = ActorSystem.Create("scadabridge", config);
|
||||
_systems.Add(system);
|
||||
return system;
|
||||
}
|
||||
|
||||
private static Task<bool> Throttle(ActorSystem from, ActorSystem to, ThrottleMode mode) =>
|
||||
((RemoteActorRefProvider)((ExtendedActorSystem)from).Provider).Transport.ManagementCommand(
|
||||
new SetThrottle(Akka.Cluster.Cluster.Get(to).SelfAddress, ThrottleTransportAdapter.Direction.Both, mode));
|
||||
|
||||
private async Task<(ActorSystem Lower, ActorSystem Higher, int LowerPort, int HigherPort)> StartPair(
|
||||
TimeSpan stableLower, TimeSpan stableHigher, bool restartWhenShutOut = true)
|
||||
{
|
||||
var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var (lowerPort, higherPort) = p1 < p2 ? (p1, p2) : (p2, p1);
|
||||
var lower = StartNode(lowerPort, higherPort, stableLower, restartWhenShutOut);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(lower, 1, TimeSpan.FromSeconds(30));
|
||||
var higher = StartNode(higherPort, lowerPort, stableHigher, restartWhenShutOut);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(higher, 2, TimeSpan.FromSeconds(30));
|
||||
return (lower, higher, lowerPort, higherPort);
|
||||
}
|
||||
|
||||
private static async Task<bool> Terminates(ActorSystem system, TimeSpan within)
|
||||
{
|
||||
var done = await Task.WhenAny(system.WhenTerminated, Task.Delay(within));
|
||||
return done == system.WhenTerminated;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Frozen-node shape: only the LOWER node downs the higher one (the higher one's window is far
|
||||
/// longer than the cut), then the network returns. The higher node — still counting the lower as
|
||||
/// a member — is refused, restarts without a tiebreak; the lower node stays Up. Then the same with
|
||||
/// roles swapped, so the single-side rule is proven independent of the address order.
|
||||
/// </summary>
|
||||
/// <param name="shutOutIsHigher">Whether the shut-out (frozen) node has the higher address.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Theory]
|
||||
[InlineData(true)]
|
||||
[InlineData(false)]
|
||||
public async Task SingleSideShutOut_ShutOutNodeRestarts_OtherStaysUp(bool shutOutIsHigher)
|
||||
{
|
||||
var quick = TimeSpan.FromSeconds(2);
|
||||
var never = TimeSpan.FromMinutes(5);
|
||||
var (lower, higher, _, _) = shutOutIsHigher
|
||||
? await StartPair(quick, never)
|
||||
: await StartPair(never, quick);
|
||||
var (survivor, shutOut) = shutOutIsHigher ? (lower, higher) : (higher, lower);
|
||||
|
||||
Assert.True(await Throttle(lower, higher, Blackhole.Instance));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(survivor, Akka.Cluster.Cluster.Get(shutOut).SelfAddress, TimeSpan.FromSeconds(30));
|
||||
Assert.Contains(Akka.Cluster.Cluster.Get(shutOut).State.Members,
|
||||
m => m.Address == Akka.Cluster.Cluster.Get(survivor).SelfAddress); // it still counts the survivor
|
||||
Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
|
||||
|
||||
Assert.True(await Terminates(shutOut, TimeSpan.FromSeconds(30)),
|
||||
"the shut-out node kept running as a second active node");
|
||||
Assert.False(await Terminates(survivor, TimeSpan.FromSeconds(5)), "the survivor restarted too");
|
||||
Assert.Equal(MemberStatus.Up, Akka.Cluster.Cluster.Get(survivor).SelfMember.Status);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A partition past the window, then healed: both sides downed and removed each other, so both
|
||||
/// are refused — exactly ONE restarts, the higher address. A fresh incarnation on the higher
|
||||
/// port (what the service supervisor starts) rejoins the survivor, and the survivor is still Up
|
||||
/// after its grace has run out.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task HealedPartition_BothShutOut_OnlyTheHigherAddressRestarts_AndRejoins()
|
||||
{
|
||||
var stable = TimeSpan.FromSeconds(2);
|
||||
var (lower, higher, lowerPort, higherPort) = await StartPair(stable, stable);
|
||||
var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress;
|
||||
var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress;
|
||||
|
||||
Assert.True(await Throttle(lower, higher, Blackhole.Instance));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30));
|
||||
Assert.False(lower.WhenTerminated.IsCompleted || higher.WhenTerminated.IsCompleted,
|
||||
"a node restarted while the partition was still in place");
|
||||
Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
|
||||
|
||||
Assert.True(await Terminates(higher, TimeSpan.FromSeconds(30)),
|
||||
"the higher-address node did not restart after the partition healed");
|
||||
Assert.False(lower.WhenTerminated.IsCompleted, "the lower-address node restarted as well");
|
||||
|
||||
// The supervisor's restart: a fresh incarnation on the same address rejoins the survivor.
|
||||
var reborn = StartNode(higherPort, lowerPort, stable);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(60));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(reborn, 2, TimeSpan.FromSeconds(60));
|
||||
|
||||
// Past the survivor's 20s grace: it saw the peer rejoin, so it stays.
|
||||
Assert.False(await Terminates(lower, TimeSpan.FromSeconds(25)), "the survivor restarted after its peer rejoined");
|
||||
Assert.Equal(2, Akka.Cluster.Cluster.Get(lower).State.Members.Count(m => m.Status == MemberStatus.Up));
|
||||
Assert.Empty(Akka.Cluster.Cluster.Get(lower).State.Unreachable);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Switch OFF: the same healed partition keeps BOTH sides running (the pre-#132 dual-active)
|
||||
/// — the positive control that the restarts above come from the feature.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task HealedPartition_SwitchOff_BothStayUp()
|
||||
{
|
||||
var stable = TimeSpan.FromSeconds(2);
|
||||
var (lower, higher, _, _) = await StartPair(stable, stable, restartWhenShutOut: false);
|
||||
var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress;
|
||||
var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress;
|
||||
|
||||
Assert.True(await Throttle(lower, higher, Blackhole.Instance));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30));
|
||||
Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
|
||||
|
||||
Assert.False(await Terminates(higher, TimeSpan.FromSeconds(15)), "a node restarted with the switch off");
|
||||
Assert.False(lower.WhenTerminated.IsCompleted, "a node restarted with the switch off");
|
||||
}
|
||||
|
||||
/// <summary>Prepares the fixture before the tests run.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public Task InitializeAsync() => Task.CompletedTask;
|
||||
|
||||
/// <summary>Releases the resources held by this fixture asynchronously.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
foreach (var s in _systems)
|
||||
{
|
||||
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user