fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132)

A node frozen past failure detection + StableAfter, or both sides of a partition that heals after each downed the other, used to stay a second active node until an operator restarted one: the peer quarantines the removed UID and Akka.Cluster 1.5.62 never acts on ThisActorSystemQuarantinedEvent. ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for leader-auto-down only) watches two signals: - S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer is still a live member here, only this side can be shut out -> restart now. - S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still runs the incarnation this node removed. A healed partition yields only S2, on both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then tiebreak. Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower stays and restarts only if still split one grace later. Removed peers are probed (Identify, 5s -> 60s back-off) since nothing else contacts them after mutual removal. Restart = Cluster.Down(self) -> existing self-down path. The Host now exits 75 after an unplanned ActorSystem termination (and sets it on the Windows service before it reports stopped): SCM recovery with failureflag 1 only restarts a service that stops with a non-zero code, and the watchdog used to exit 0. Shipped default strategy stays auto-down. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:57:42 -05:00
parent a633005f4f
commit 952db02aab
14 changed files with 1646 additions and 10 deletions
@@ -111,7 +111,9 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
 ### Downed-node recovery
-`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).
+`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits **with code 75** (`AkkaHostedService.SelfDownExitCode`; set by `Program` after the host stops, and on the Windows service itself before it reports `SERVICE_STOPPED`) and the service supervisor (docker `restart: unless-stopped`, Windows service recovery with `failureflag 1`, which only acts on a non-zero exit code) restarts it as a fresh incarnation (`AkkaHostedService.GetOrCreateActorSystem`).
 Under `leader-auto-down` the same path restarts a node that its peer has **shut out** — downed, removed and quarantined it (a frozen node coming back), or both sides of a healed partition, where the higher address restarts. See `Component-ClusterInfrastructure.md` → *Shut-out restart* (ScadaBridge#132).
 **Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.
@@ -104,3 +104,36 @@ Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the prove
 - **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that
  both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged).
 - **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs.
 ## 7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132)
 The #116 rig drill found a gap that `AutoDowning` and its port share: a node frozen past failure detection plus the window
 comes back as a **permanent second active node**, and the same happens to both sides of a partition that heals after each
 side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote
 publishes `ThisActorSystemQuarantinedEvent` on the stale node, and Akka.Cluster never subscribes to it.
 **Owner decision (#132):** under `leader-auto-down`, a node that is shut out restarts itself; when both sides are shut out,
 exactly one restarts, by a deterministic tiebreak. Implemented as `ShutOutRestart` (Host, a child of the downing actor, so
 `LeaderAutoDownBase` stays a line-for-line port), behind `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default
 `true` for this strategy only) and `ShutOutGrace` (default 60 s). The shipped default strategy stays `auto-down`.
 - **Two signals, confirmed in-process.** S1 `ThisActorSystemQuarantinedEvent` (the frozen node). S2 an `AssociationErrorEvent`
  whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed
  partition produces **only S2, on both sides** — each side aborts the other's handshake with its own refused UID before a
  quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with
  `Identify` (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it.
 - **Decision.** S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this
  node removed the peer: tiebreak. S2: wait `ShutOutGrace` (a frozen peer restarts itself meanwhile), then tiebreak if still
  split. Tiebreak: **higher address restarts**, the lower stays and restarts only if the peer is still split from it one
  more grace later.
 - **Why address, not age:** see `Component-ClusterInfrastructure.md` → *Shut-out restart*. In short, both nodes hold both
  addresses always; the peer's up-number is history that can differ after the split.
 - **Exit code.** The restart is the existing self-down path, and it exposed a latent gap: the watchdog's `StopApplication()`
  exited 0, and Windows service recovery with `failureflag 1` acts only on a non-zero exit. The Host now exits 75 after an
  unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code.
 - **Contract change for #116 gate 4:** the partition still produces dual-active *while it lasts*; after it heals, one side
  (the higher address) restarts and the pair converges with no operator action.
  `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds (it never heals).
 - **Tests:** `ShutOutRestartTests` (unit), `ShutOutRestartScenarioTests` (two real nodes: freeze shape both address orders,
  healed partition, switch off), `UnexpectedTerminationTests`, `HoconBuilderTests`, `ClusterOptionsValidatorTests`.
  Rig evidence: `2026-11-12-leader-auto-down-rig-drill.md` §7.
@@ -104,6 +104,24 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
 Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
 #### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
 Without it, a node that was **frozen** (VM pause, long GC stall, `docker pause`) for longer than failure detection plus `StableAfter` comes back as a **second, permanent active node**: its peer has already downed and removed it, quarantining its UID, and Akka.Cluster 1.5.62 never tells the frozen node — so it downs the peer in its own one-node view. Both sides of a **real partition that heals** after each side downed the other end the same way. This happens under `auto-down` too (Akka core behaviour); the in-repo provider is where it is fixed.
 With `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` on (**the default whenever `leader-auto-down` is selected**), the `ShutOutRestart` actor (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/ShutOutRestart.cs`, a child of the downing actor) restarts the stale side:
 | Signal (Akka.NET 1.5.62) | Who sees it | Decision |
 |---|---|---|
 | **S1** `ThisActorSystemQuarantinedEvent` — the peer has quarantined *this* node — while the peer is still a live member in this node's view | the frozen node | **restart now**; no tiebreak (this node has not removed the peer, so the peer cannot be shut out by it) |
 | **S1** after this node has removed the peer too | either side of a split | **tiebreak now** |
 | **S2** `AssociationErrorEvent` "remote system has a UID that has been quarantined" — the peer still runs the incarnation *this* node removed | both sides of a healed partition; the survivor of a freeze | wait `ShutOutGrace` (a frozen peer restarts itself on S1 meanwhile); still split at the end ⇒ **tiebreak** |
 **Tiebreak: the higher address (host, then port — Akka's `Member.AddressOrdering`) restarts; the lower stays.** The addresses are in every signal and in both nodes' seed lists, so both nodes compute the same answer without talking. Member age (up-number) was rejected: by the time a signal fires both sides have removed each other, so the peer's up-number exists only as remembered history that can differ (an unconverged join, a restart since), and a disagreement means both restart or neither. The lower address is also the Akka leader of the re-formed pair and the bootstrap guard's preferred founder. The lower node gives its peer one more `ShutOutGrace` and restarts only if the peer has neither rejoined nor gone quiet by then (the single-sided case where the stale side happened to be the lower address).
 After both sides have removed each other neither sends the other anything, so a healed partition would stay silent: the actor **probes** (an `Identify` to `/system/cluster/core/daemon`) every peer it saw removed after a Down, backing off 5 s → 60 s, until that address is a member again. Each probe of a dead peer logs one Akka association warning.
 "Restart" is the existing self-down path (below): `Cluster.Down(self)` ⇒ the `ActorSystem` terminates ⇒ the process exits **75** ⇒ the supervisor restarts it and it rejoins as standby. Every decision logs a `[leader-auto-down]` line: `shut out by <peer> … tiebreak not needed — this node RESTARTS`, `split from <peer> … Giving it <grace> …`, `Tiebreak says RESTART|STAY`, `rejoined … stays`. Settings: `RestartWhenShutOut` (bool, default `true`) and `ShutOutGrace` (default `60s`, must be positive, must cover the peer's worst-case shutdown). Off restores the pre-#132 behaviour (dual-active until an operator restarts one side). Proven by `ShutOutRestartTests` (unit) and `ShutOutRestartScenarioTests` (real two-node clusters, freeze shape and healed partition); rig evidence in `docs/plans/2026-11-12-leader-auto-down-rig-drill.md` §7.
 ### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
 - **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
@@ -116,8 +134,8 @@ Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `
 When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
 1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
-2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
+2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits — with code 75** (`AkkaHostedService.SelfDownExitCode`, `EX_TEMPFAIL`; ScadaBridge#132). Under the Windows service lifetime the code is also set on the service (`ServiceBase.ExitCode`) before the stop, so the SCM sees it.
-3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
+3. The service supervisor restarts it — docker `restart: unless-stopped` (any exit code), or Windows service recovery actions (`sc.exe failure … restart/…` **plus `sc.exe failureflag <svc> 1`**, both set by the wonder install script `deploy/wonder-app-vd03/install.ps1`, which is kept outside git). With `failureflag 1` the SCM runs the recovery actions for a service that stops with a **non-zero** exit code; before #132 the watchdog exit reported 0, which the SCM treats as a clean stop. (ScadaBridge runs as native .NET Windows services; NSSM wraps only the LDAP directory.)
 4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.
 ### Seed Node Ordering
@@ -144,6 +144,61 @@ public class ClusterOptions
    /// <c>docs/plans/2026-11-09-tls-production-profile-design.md</c>.
    /// </summary>
    public ClusterRemotingTlsOptions RemotingTls { get; set; } = new();
    /// <summary>
    /// Settings that apply only when <see cref="SplitBrainResolverStrategy"/> is
    /// <c>leader-auto-down</c> (<c>ScadaBridge:Cluster:LeaderAutoDown</c>, ScadaBridge#132).
    /// Inert under every other strategy: <c>AkkaHostedService.BuildHocon</c> emits none of it.
    /// </summary>
    public LeaderAutoDownOptions LeaderAutoDown { get; set; } = new();
 }
 /// <summary>
 /// <c>leader-auto-down</c>-only settings (ScadaBridge#132): what a node does when it learns that
 /// its pair peer has <b>shut it out</b> — removed and quarantined it, so it can never talk to that
 /// peer again until it restarts.
 /// </summary>
 /// <remarks>
 /// <para>
 /// <b>Why it exists.</b> A node that was frozen (VM pause, GC stall, <c>docker pause</c>) past the
 /// failure-detection window plus <see cref="ClusterOptions.StableAfter"/> is downed and removed by
 /// its peer. When it resumes, every association is refused with "the remote system has quarantined
 /// this system" — Akka.Remote publishes <c>ThisActorSystemQuarantinedEvent</c>, but Akka.Cluster
 /// 1.5.62 never acts on it, so the frozen node downs the peer in its own one-node view and becomes
 /// a second, permanent active node. The same thing happens to <b>both</b> nodes after a real
 /// network partition heals.
 /// </para>
 /// <para>
 /// <b>What it does</b> (with <see cref="RestartWhenShutOut"/> on): a node that is shut out by its
 /// peer while it still counts that peer as a member restarts itself (it is the stale side). When it
 /// has already removed the peer too — both sides shut each other out — a deterministic tiebreak
 /// both nodes compute identically picks ONE side: the node with the <b>higher</b> address
 /// (host, then port; Akka's own member address ordering) restarts, the lower stays. The lower node
 /// restarts too only if it is still being shut out after <see cref="ShutOutGrace"/>. "Restart" is
 /// the existing self-down path: <c>Cluster.Down(self)</c>, the <c>ActorSystem</c> terminates, the
 /// Host exits non-zero and the service supervisor restarts it as a fresh incarnation that rejoins
 /// as standby. Implemented by <c>ShutOutRestart</c> in the Host.
 /// </para>
 /// </remarks>
 public sealed class LeaderAutoDownOptions
 {
    /// <summary>
    /// Whether a node shut out by its pair peer restarts itself (see the class remarks). Default
    /// <see langword="true"/> — on whenever <c>leader-auto-down</c> is selected. Set
    /// <see langword="false"/> to get the old behaviour back: a frozen node or a healed partition
    /// leaves two active nodes until an operator restarts one.
    /// </summary>
    public bool RestartWhenShutOut { get; set; } = true;
    /// <summary>
    /// How long the tiebreak WINNER (the lower-address node) waits for its peer to restart before
    /// it re-checks. At the end of the grace it restarts only if the peer is still not a member AND
    /// it was shut out again in the last few seconds — i.e. the peer never restarted, so this node
    /// is the stale side after all. Must cover the peer's worst-case shut-down time (its old
    /// incarnation keeps refusing until its <c>ActorSystem</c> has terminated). Default 60s.
    /// Validated <c>&gt; 0</c> at boot when the feature is on.
    /// </summary>
    public TimeSpan ShutOutGrace { get; set; } = TimeSpan.FromSeconds(60);
 }
 /// <summary>
@@ -74,6 +74,17 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
            $"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
            "declared unreachable before a heartbeat can arrive.");
        // ScadaBridge#132: the shut-out restart grace is only consulted under leader-auto-down
        // with the feature on; a zero/negative grace would make the tiebreak winner restart on
        // the spot, i.e. BOTH nodes of a healed partition restart together.
        var isLeaderAutoDown = string.Equals(
            options.SplitBrainResolverStrategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase);
        var leaderAutoDown = options.LeaderAutoDown ?? new LeaderAutoDownOptions();
        builder.RequireThat(
            !isLeaderAutoDown || !leaderAutoDown.RestartWhenShutOut || leaderAutoDown.ShutOutGrace > TimeSpan.Zero,
            "ClusterOptions.LeaderAutoDown.ShutOutGrace must be a positive duration when "
            + "LeaderAutoDown.RestartWhenShutOut is on (ScadaBridge#132).");
        // DownIfAlone is a keep-oldest knob; under (leader-)auto-down each side downs the
        // unreachable peer regardless, so the flag is inert and any value is fine.
        var isKeepOldest = string.Equals(
@@ -34,8 +34,20 @@ public class AkkaHostedService : IHostedService
    private readonly CommunicationOptions _communicationOptions;
    private readonly ILogger<AkkaHostedService> _logger;
    private readonly IHostApplicationLifetime? _appLifetime;
    private readonly IHostLifetime? _hostLifetime;
    private ActorSystem? _actorSystem;
    /// <summary>
    /// Process exit code after an ActorSystem termination outside host shutdown (a self-down,
    /// or a ScadaBridge#132 shut-out restart): 75, <c>EX_TEMPFAIL</c> — "temporary failure, try
    /// again". It must be NON-zero: docker <c>restart: unless-stopped</c> restarts on any exit, but
    /// Windows service recovery (<c>sc.exe failure … restart</c> with <c>failureflag 1</c>, set by
    /// <c>deploy/wonder-app-vd03/install.ps1</c>) applies its restart actions to a service that
    /// reports <c>SERVICE_STOPPED</c> only when the reported exit code is not 0. A clean
    /// <c>StopApplication()</c> alone reports 0, so the node would stay stopped.
    /// </summary>
    public const int SelfDownExitCode = 75;
    /// <summary>
    /// Flips to <c>true</c> the moment <see cref="StopAsync"/> begins, so the
    /// <see cref="ActorSystem.WhenTerminated"/> watchdog can tell an
@@ -84,14 +96,17 @@ public class AkkaHostedService : IHostedService
    /// <param name="communicationOptions">The communication configuration options.</param>
    /// <param name="logger">The logger instance.</param>
    /// <param name="appLifetime">The host application lifetime; optional so existing construction sites (tests, DI variants) keep working without it.</param>
    /// <param name="hostLifetime">The host lifetime; when it is the Windows service lifetime, the self-down exit code is reported to the SCM through it. Optional.</param>
    public AkkaHostedService(
        IServiceProvider serviceProvider,
        IOptions<NodeOptions> nodeOptions,
        IOptions<ClusterOptions> clusterOptions,
        IOptions<CommunicationOptions> communicationOptions,
        ILogger<AkkaHostedService> logger,
-        IHostApplicationLifetime? appLifetime = null)
+        IHostApplicationLifetime? appLifetime = null,
        IHostLifetime? hostLifetime = null)
    {
        _hostLifetime = hostLifetime;
        _serviceProvider = serviceProvider;
        _nodeOptions = nodeOptions.Value;
        _clusterOptions = clusterOptions.Value;
@@ -212,8 +227,11 @@ public class AkkaHostedService : IHostedService
            {
                if (_stopRequested) return;
                _logger.LogCritical(
-                    "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down). "
+                    "ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / "
-                    + "Stopping the host process so the service supervisor restarts this node as a fresh incarnation.");
+                    + "leader-auto-down shut-out restart). Stopping the host process with exit code {ExitCode} so the "
                    + "service supervisor restarts this node as a fresh incarnation.", SelfDownExitCode);
                _terminatedOutsideHostShutdown = true;
                MarkServiceSelfDownExit(_hostLifetime);
                _appLifetime?.StopApplication();
            }, TaskContinuationOptions.ExecuteSynchronously);
@@ -223,6 +241,37 @@ public class AkkaHostedService : IHostedService
        }
    }
    /// <summary>
    /// <see langword="true"/> once the <see cref="ActorSystem"/> has terminated outside
    /// <see cref="StopAsync"/> (a self-down or a shut-out restart). <c>Program</c> reads it after
    /// <c>RunAsync</c> returns and exits with <see cref="SelfDownExitCode"/>; it is a flag rather than a
    /// direct write of the process exit code so test hosts that terminate a system never inherit
    /// a non-zero process exit code.
    /// </summary>
    public bool TerminatedOutsideHostShutdown => _terminatedOutsideHostShutdown;
    private volatile bool _terminatedOutsideHostShutdown;
    /// <summary>
    /// When the host runs as a Windows service, sets <see cref="SelfDownExitCode"/> as the exit code
    /// the service reports to the SCM when it stops — the value Windows service recovery
    /// (<c>failureflag 1</c>) keys its restart on. Must run BEFORE <c>StopApplication()</c>: the
    /// Windows service lifetime reports <c>SERVICE_STOPPED</c> during host shutdown, before
    /// <c>Program</c> can set the process exit code.
    /// </summary>
    /// <param name="hostLifetime">The host lifetime; the Windows service lifetime is a <c>ServiceBase</c>.</param>
    /// <returns><see langword="true"/> when the exit code was set on a Windows service lifetime.</returns>
    internal static bool MarkServiceSelfDownExit(IHostLifetime? hostLifetime)
    {
        if (OperatingSystem.IsWindows() && hostLifetime is System.ServiceProcess.ServiceBase service)
        {
            service.ExitCode = SelfDownExitCode;
            return true;
        }
        return false;
    }
    /// <summary>
    /// Builds the Akka HOCON configuration document. Every interpolated value is
    /// routed through <see cref="QuoteHocon"/> (string values) so a hostname,
@@ -313,8 +362,12 @@ public class AkkaHostedService : IHostedService
        // The leader-auto-down duration sits outside akka.cluster so Akka never mistakes it
        // for one of its own (deprecated) auto-down keys. Only emitted for that strategy, so
        // the auto-down / keep-oldest documents are byte-identical to before.
        // ScadaBridge#132: the shut-out restart switch and grace ride in the same block.
        var shutOut = clusterOptions.LeaderAutoDown ?? new LeaderAutoDownOptions();
        var leaderAutoDownBlock = string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase)
            ? $"{LeaderAutoDowningProvider.UnreachableAfterPath} = {DurationHocon(clusterOptions.StableAfter)}\n"
              + $"{ShutOutRestartSettings.EnabledPath} = {(shutOut.RestartWhenShutOut ? "on" : "off")}\n"
              + $"{ShutOutRestartSettings.GracePath} = {DurationHocon(shutOut.ShutOutGrace)}\n"
            : string.Empty;
        // Akka.Remote TLS (ScadaBridge#64, opt-in). Emitted inside dot-netty.tcp only when
@@ -88,7 +88,8 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider
                    $"LeaderAutoDowningProvider selected but '{UnreachableAfterPath}' not set");
            }
-            return LeaderAutoDown.Props(after.Value, _cluster);
+            return LeaderAutoDown.Props(
                after.Value, _cluster, ShutOutRestartSettings.FromConfig(_system.Settings.Config));
        }
    }
@@ -118,21 +119,29 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider
 /// </summary>
 internal sealed class LeaderAutoDown : LeaderAutoDownBase
 {
    /// <summary>Child name of the <see cref="ShutOutRestart"/> actor.</summary>
    internal const string ShutOutRestartName = "shut-out-restart";
    private readonly Akka.Cluster.Cluster _cluster;
    private readonly ShutOutRestartSettings? _shutOut;
    /// <summary>Props for the downing actor.</summary>
    /// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
    /// <param name="cluster">The cluster extension.</param>
-    public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) =>
+    /// <param name="shutOut">Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling.</param>
-        Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster));
+    /// <returns>The props.</returns>
    public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut = null) =>
        Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster, shutOut));
    /// <summary>Creates the actor.</summary>
    /// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
    /// <param name="cluster">The cluster extension.</param>
-    public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster)
+    /// <param name="shutOut">Shut-out restart settings (ScadaBridge#132); null or disabled = no shut-out handling.</param>
    public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster, ShutOutRestartSettings? shutOut)
        : base(unreachableAfter)
    {
        _cluster = cluster;
        _shutOut = shutOut;
    }
    /// <inheritdoc />
@@ -145,6 +154,22 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase
    protected override void PreStart()
    {
        _cluster.Subscribe(Self, typeof(ClusterEvent.IClusterDomainEvent));
        // ScadaBridge#132: the shut-out restart is a separate child so the downing logic above
        // stays a line-for-line port of Akka's AutoDownBase. A child, not a sibling, so it lives
        // and dies with this provider's actor and exists only under leader-auto-down.
        if (_shutOut is { Enabled: true })
        {
            Context.ActorOf(ShutOutRestart.Props(_shutOut, _cluster), ShutOutRestartName);
        }
        else
        {
            Context.GetLogger().Info(
                "[leader-auto-down] shut-out restart is OFF for {0} (ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut = false): "
                + "a node its peer has removed and quarantined keeps running as a second active node until it is restarted",
                _cluster.SelfAddress);
        }
        base.PreStart();
    }
@@ -0,0 +1,674 @@
 using Akka.Actor;
 using Akka.Cluster;
 using Akka.Configuration;
 using Akka.Event;
 using Akka.Remote;
 namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
 /// <summary>
 /// Settings for <see cref="ShutOutRestart"/>, read from the <c>scadabridge.cluster.leader-auto-down</c>
 /// HOCON block that <see cref="AkkaHostedService.BuildHocon"/> emits from
 /// <c>ScadaBridge:Cluster:LeaderAutoDown</c> (ScadaBridge#132).
 /// </summary>
 /// <param name="Enabled">Whether a node that is shut out by its peer restarts itself.</param>
 /// <param name="Grace">How long a peer gets to restart itself before this node acts (see <see cref="ShutOutRestart"/>).</param>
 /// <param name="ProbeInterval">First interval between probes of a removed peer, and the fixed interval while a grace runs.</param>
 /// <param name="ProbeMaxInterval">Cap for the probe back-off while no grace runs.</param>
 internal sealed record ShutOutRestartSettings(
    bool Enabled, TimeSpan Grace, TimeSpan ProbeInterval, TimeSpan ProbeMaxInterval)
 {
    /// <summary>HOCON switch: <c>on</c>/<c>off</c>. Missing = on.</summary>
    public const string EnabledPath = "scadabridge.cluster.leader-auto-down.restart-when-shut-out";
    /// <summary>HOCON duration for <see cref="Grace"/>. Missing = 60s.</summary>
    public const string GracePath = "scadabridge.cluster.leader-auto-down.shut-out-grace";
    /// <summary>HOCON duration for <see cref="ProbeInterval"/>. Missing = 5s. Not bound from appsettings (tests only).</summary>
    public const string ProbeIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-interval";
    /// <summary>HOCON duration for <see cref="ProbeMaxInterval"/>. Missing = 60s. Not bound from appsettings (tests only).</summary>
    public const string ProbeMaxIntervalPath = "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval";
    /// <summary>
    /// How recent the last shut-out evidence must be, when a grace ends, for the split to count as
    /// still in place: three probe intervals, so a missed probe or two does not flip the verdict.
    /// </summary>
    public TimeSpan RecentWindow => ProbeInterval * 3;
    /// <summary>Reads the settings, defaulting every missing key.</summary>
    /// <param name="config">The actor system's configuration.</param>
    /// <returns>The settings.</returns>
    public static ShutOutRestartSettings FromConfig(Config config)
    {
        var enabled = !config.HasPath(EnabledPath) || config.GetBoolean(EnabledPath, true);
        return new ShutOutRestartSettings(
            enabled,
            Duration(config, GracePath, TimeSpan.FromSeconds(60)),
            Duration(config, ProbeIntervalPath, TimeSpan.FromSeconds(5)),
            Duration(config, ProbeMaxIntervalPath, TimeSpan.FromSeconds(60)));
    }
    private static TimeSpan Duration(Config config, string path, TimeSpan fallback) =>
        config.HasPath(path) ? config.GetTimeSpan(path, fallback, true) : fallback;
 }
 /// <summary>
 /// ScadaBridge#132: makes a node that is split from its pair peer — the peer has downed and removed
 /// it, or each has removed the other — restart itself, so a frozen node or a healed partition does
 /// not leave two active nodes until an operator acts. Runs as a child of the
 /// <see cref="LeaderAutoDown"/> downing actor, so it exists only under <c>leader-auto-down</c>, and
 /// only when <c>ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut</c> is on (the default for
 /// that strategy).
 /// <para>
 /// <b>The signals</b> (Akka.NET 1.5.62, read from the decompiled assemblies and confirmed by the
 /// in-process scenario tests). When a node removes a downed member, its cluster quarantines that
 /// member's UID. After that, each association between the two incarnations fails, and which side
 /// learns what depends on who still counts whom as a member:
 /// <list type="bullet">
 /// <item><b>S1 — "the peer has quarantined THIS node".</b> Akka.Remote publishes
 /// <see cref="ThisActorSystemQuarantinedEvent"/> (<c>EndpointManager</c>, on an
 /// <c>InvalidAssociation</c> whose disassociate reason is <c>Quarantined</c>). Seen by a node that
 /// still talks to a peer which has removed it: the frozen node after <c>docker pause</c>. Akka.Cluster
 /// 1.5.62 never subscribes to it — <c>ClusterCoreDaemon</c> handles only the opposite
 /// <c>QuarantinedEvent</c> — which is the root cause of #132.</item>
 /// <item><b>S2 — "the peer is still running the incarnation THIS node removed".</b> An outbound
 /// handshake that meets a UID this node has quarantined is aborted (<c>ProtocolStateActor</c>,
 /// <c>ForbiddenUidReason</c>: "The remote system has a UID that has been quarantined"), surfacing as
 /// an <see cref="AssociationErrorEvent"/>. Seen by both sides of a partition that healed after each
 /// side downed the other — neither of them sees S1, because each aborts the other's handshake first —
 /// and by the survivor of a freeze when it probes the frozen node.</item>
 /// </list>
 /// </para>
 /// <para>
 /// <b>The decision.</b>
 /// <list type="number">
 /// <item><b>S1 while the peer is still a live member in this node's view</b> (Joining, WeaklyUp, Up
 /// or Leaving): this node has not removed the peer, so it has not quarantined it, so the peer
 /// cannot be shut out by this node — only this side is. This is the frozen node. <b>Restart
 /// now</b>; no tiebreak is needed.</item>
 /// <item><b>S1 after this node has removed the peer too</b>: both sides are shut out.
 /// <b>Tiebreak now</b> (below).</item>
 /// <item><b>S2</b>: the peer's old incarnation is alive. It may be a frozen node about to restart
 /// itself on S1 — so this node first gives it <see cref="ShutOutRestartSettings.Grace"/>. If, at
 /// the end, the peer has neither rejoined nor gone quiet, both sides are shut out: <b>tiebreak</b>.
 /// </item>
 /// <item><b>Tiebreak</b>: the node with the <b>higher</b> address restarts now. The lower one stays
 /// and gives the peer one more <see cref="ShutOutRestartSettings.Grace"/>; it restarts only if the
 /// peer has still not rejoined and is still producing evidence at the end — the single-sided case
 /// where the stale side happened to be the lower address.</item>
 /// </list>
 /// "Quiet" means no S1/S2 from that peer for <see cref="ShutOutRestartSettings.RecentWindow"/>: the
 /// old incarnation is gone (restarting, or dead), so there is nothing left to split from.
 /// </para>
 /// <para>
 /// <b>Why the address tiebreak, not member age.</b> Both nodes must reach the same answer without
 /// talking to each other, from state they already hold. The two addresses are in every signal and
 /// in both nodes' seed lists, so the comparison is identical on both sides by construction. Age
 /// (up-number) is not: by the time a signal fires both sides have removed each other, so the peer's
 /// up-number survives only as remembered history, which differs if a join never converged before
 /// the split or either side has restarted since. A disagreement there means both restart or neither
 /// does — exactly what the tiebreak exists to prevent. The ordering is Akka's own
 /// <see cref="Member.AddressOrdering"/> (host, then port), which also picks the Akka leader of the
 /// re-formed pair, and matches the bootstrap guard, where the lower address is the preferred founder.
 /// </para>
 /// <para>
 /// <b>Probing.</b> After both sides have removed each other neither sends the other anything, so a
 /// healed partition would stay silent and neither signal would ever fire. This actor therefore
 /// sends an <see cref="Identify"/> to every peer it saw removed after being downed, with a back-off
 /// from <see cref="ShutOutRestartSettings.ProbeInterval"/> to
 /// <see cref="ShutOutRestartSettings.ProbeMaxInterval"/> (fixed at the short interval while a grace
 /// runs), until that address is a member again. A probe of a restarted incarnation (new UID) is
 /// harmless — the new incarnation is joining anyway; a probe of the old incarnation produces S2
 /// here and S1 or S2 there.
 /// </para>
 /// <para>
 /// <b>The restart</b> reuses the existing self-down path: <c>Cluster.Down(self)</c> ⇒
 /// <c>run-coordinated-shutdown-when-down</c> ⇒ the <c>ActorSystem</c> terminates ⇒ the Host's
 /// <c>WhenTerminated</c> watchdog exits the process with <see cref="AkkaHostedService.SelfDownExitCode"/>
 /// ⇒ the service supervisor (docker <c>restart: unless-stopped</c>, Windows service recovery with
 /// <c>failureflag 1</c>) starts a fresh incarnation that rejoins its peer as standby. If the system
 /// has not terminated <see cref="FallbackShutdownAfter"/> after the Down, <c>CoordinatedShutdown</c>
 /// is run directly.
 /// </para>
 /// </summary>
 internal sealed class ShutOutRestart : ShutOutRestartBase
 {
    /// <summary>How long after <c>Cluster.Down(self)</c> the direct <c>CoordinatedShutdown</c> fallback runs.</summary>
    internal static readonly TimeSpan FallbackShutdownAfter = TimeSpan.FromSeconds(20);
    private readonly Akka.Cluster.Cluster _cluster;
    private ICancelable? _fallback;
    /// <summary>Props for the actor.</summary>
    /// <param name="settings">The settings.</param>
    /// <param name="cluster">The cluster extension.</param>
    /// <returns>The props.</returns>
    public static Props Props(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster) =>
        Akka.Actor.Props.Create(() => new ShutOutRestart(settings, cluster));
    /// <summary>Creates the actor.</summary>
    /// <param name="settings">The settings.</param>
    /// <param name="cluster">The cluster extension.</param>
    public ShutOutRestart(ShutOutRestartSettings settings, Akka.Cluster.Cluster cluster)
        : base(settings)
    {
        _cluster = cluster;
    }
    /// <inheritdoc />
    public override Address SelfAddress => _cluster.SelfAddress;
    /// <inheritdoc />
    public override IScheduler Scheduler => Context.System.Scheduler;
    /// <inheritdoc />
    protected override IEnumerable<Address> SeedNodes => _cluster.Settings.SeedNodes;
    /// <inheritdoc />
    protected override void PreStart()
    {
        _cluster.Subscribe(Self, ClusterEvent.SubscriptionInitialStateMode.InitialStateAsSnapshot,
            typeof(ClusterEvent.IMemberEvent));
        Context.System.EventStream.Subscribe(Self, typeof(ThisActorSystemQuarantinedEvent));
        Context.System.EventStream.Subscribe(Self, typeof(AssociationErrorEvent));
        base.PreStart();
    }
    /// <inheritdoc />
    protected override void PostStop()
    {
        _cluster.Unsubscribe(Self);
        Context.System.EventStream.Unsubscribe(Self);
        _fallback?.Cancel();
        base.PostStop();
    }
    /// <inheritdoc />
    protected override void RestartSelf()
    {
        _cluster.Down(_cluster.SelfAddress);
        _fallback = Scheduler.ScheduleTellOnceCancelable(FallbackShutdownAfter, Self, FallbackShutdown.Instance, Self);
    }
    /// <inheritdoc />
    protected override void Probe(Address peer)
    {
        Context.ActorSelection(new RootActorPath(peer) / "system" / "cluster" / "core" / "daemon")
            .Tell(new Identify(ProbeMessageId), Self);
    }
    /// <inheritdoc />
    protected override void OnFallbackShutdown()
    {
        if (!Context.System.WhenTerminated.IsCompleted)
        {
            Log.Warning(
                "[leader-auto-down] {0} was downed for a shut-out restart {1} ago but its actor system is still running; running CoordinatedShutdown directly",
                SelfAddress, FallbackShutdownAfter);
            CoordinatedShutdown.Get(Context.System).Run(CoordinatedShutdown.ClusterDowningReason.Instance);
        }
    }
 }
 /// <summary>
 /// The cluster-free decision logic of <see cref="ShutOutRestart"/>, split out (as
 /// <see cref="LeaderAutoDownBase"/> is) so the tiebreak and the single- and both-sides cases can be
 /// unit tested by feeding events straight to the actor. Every decision is logged with the
 /// <c>[leader-auto-down]</c> prefix; the restart / stay verdicts at Warning.
 /// </summary>
 internal abstract class ShutOutRestartBase : UntypedActor
 {
    /// <summary>The <see cref="Identify.MessageId"/> carried by every probe.</summary>
    internal const string ProbeMessageId = "scadabridge-shut-out-probe";
    /// <summary>Timer: probe <see cref="Peer"/> again.</summary>
    /// <param name="Peer">The removed peer to probe.</param>
    internal sealed record ProbeTick(Address Peer);
    /// <summary>Timer: the grace for <see cref="Peer"/> has ended.</summary>
    /// <param name="Peer">The peer this node is split from.</param>
    internal sealed record GraceExpired(Address Peer);
    /// <summary>Timer: the direct-shutdown fallback after a restart was requested.</summary>
    internal sealed class FallbackShutdown
    {
        /// <summary>The singleton instance.</summary>
        public static readonly FallbackShutdown Instance = new();
        private FallbackShutdown()
        {
        }
    }
    /// <summary>A peer identity independent of the transport scheme (the throttler adapter rewrites it).</summary>
    private readonly record struct PeerKey(string System, string? Host, int? Port)
    {
        public static PeerKey Of(Address address) => new(address.System, address.Host, address.Port);
    }
    private sealed class ProbeState
    {
        public required Address Address { get; init; }
        public TimeSpan NextInterval { get; set; }
        public ICancelable? Timer { get; set; }
    }
    /// <summary>Phase of a detected split with one peer.</summary>
    private enum SplitPhase
    {
        /// <summary>S2 seen: the peer's old incarnation is alive; it gets a grace to restart itself.</summary>
        Suspect,
        /// <summary>Tiebreak said stay: the peer is expected to restart; it gets a grace to do so.</summary>
        Winner,
    }
    private sealed class SplitState
    {
        public required SplitPhase Phase { get; set; }
        public required ICancelable Timer { get; set; }
        public DateTime LastEvidenceUtc { get; set; }
    }
    private readonly ShutOutRestartSettings _settings;
    private readonly Dictionary<PeerKey, Member> _members = new();
    private readonly HashSet<PeerKey> _knownPeers = new();
    private readonly HashSet<PeerKey> _ignoredLogged = new();
    private readonly Dictionary<PeerKey, ProbeState> _probing = new();
    private readonly Dictionary<PeerKey, SplitState> _splits = new();
    private bool _restarting;
    /// <summary>Creates the base logic.</summary>
    /// <param name="settings">The settings.</param>
    protected ShutOutRestartBase(ShutOutRestartSettings settings)
    {
        _settings = settings;
    }
    /// <summary>Decision log.</summary>
    protected ILoggingAdapter Log { get; } = Context.GetLogger();
    /// <summary>This node's address.</summary>
    public abstract Address SelfAddress { get; }
    /// <summary>The scheduler used for the probe and grace timers.</summary>
    public abstract IScheduler Scheduler { get; }
    /// <summary>The configured seed nodes; peers of this pair even before they were ever seen as members.</summary>
    protected abstract IEnumerable<Address> SeedNodes { get; }
    /// <summary>The clock; overridable for tests.</summary>
    protected virtual DateTime UtcNow => DateTime.UtcNow;
    /// <summary>Restarts this node (called at most once).</summary>
    protected abstract void RestartSelf();
    /// <summary>Sends one probe to <paramref name="peer"/>.</summary>
    /// <param name="peer">The removed peer.</param>
    protected abstract void Probe(Address peer);
    /// <summary>Runs the direct-shutdown fallback, if the restart has not taken effect.</summary>
    protected virtual void OnFallbackShutdown()
    {
    }
    /// <summary>
    /// Whether an association failure is S2: the handshake met a UID this node has quarantined.
    /// Akka.Remote 1.5.62 carries no typed reason outward — the protocol layer's
    /// <c>ForbiddenUidReason</c> becomes "The remote system has a UID that has been quarantined.
    /// Association aborted." and the endpoint layer's <c>HopelessAssociation</c> says "The remote
    /// system [..] has a UID [..] that has been quarantined." — so the cause chain is matched on that
    /// wording. Pinned by the in-process scenario tests, which fail if an Akka upgrade rewords it.
    /// </summary>
    /// <param name="cause">The association error's cause.</param>
    /// <returns><see langword="true"/> when the failure is a refused, quarantined UID.</returns>
    internal static bool IsQuarantinedUidRefusal(Exception? cause)
    {
        for (var depth = 0; cause is not null && depth < 8; depth++)
        {
            var message = cause.Message;
            if (message.Contains("has a UID", StringComparison.Ordinal)
                && message.Contains("that has been quarantined", StringComparison.Ordinal))
            {
                return true;
            }
            if (cause is AggregateException aggregate && aggregate.InnerExceptions.Any(IsQuarantinedUidRefusal))
            {
                return true;
            }
            cause = cause.InnerException;
        }
        return false;
    }
    /// <inheritdoc />
    protected override void PreStart()
    {
        foreach (var seed in SeedNodes)
        {
            AddKnownPeer(seed);
        }
        Log.Info(
            "[leader-auto-down] shut-out restart is ON for {0}: a node split from its pair peer restarts itself — at once if only it "
            + "was removed, otherwise the higher address restarts after a {1} grace (ScadaBridge#132)",
            SelfAddress, _settings.Grace);
    }
    /// <inheritdoc />
    protected override void PostStop()
    {
        CancelAllTimers();
    }
    /// <inheritdoc />
    protected override void OnReceive(object message)
    {
        switch (message)
        {
            case ClusterEvent.CurrentClusterState state:
                _members.Clear();
                foreach (var member in state.Members)
                {
                    TrackMember(member);
                }
                break;
            case ClusterEvent.MemberRemoved removed:
                _members.Remove(PeerKey.Of(removed.Member.Address));
                if (removed.PreviousStatus == MemberStatus.Down && !IsSelf(removed.Member.Address))
                {
                    StartProbing(removed.Member.Address);
                }
                break;
            case ClusterEvent.IMemberEvent memberEvent:
                TrackMember(memberEvent.Member);
                break;
            case ThisActorSystemQuarantinedEvent shutOut:
                OnShutOutByPeer(shutOut.RemoteAddress);
                break;
            case AssociationErrorEvent error:
                if (IsQuarantinedUidRefusal(error.Cause))
                {
                    OnPeerStillRunningRemovedIncarnation(error.RemoteAddress);
                }
                break;
            case ProbeTick tick:
                OnProbeTick(tick.Peer);
                break;
            case GraceExpired expired:
                OnGraceExpired(expired.Peer);
                break;
            case FallbackShutdown:
                OnFallbackShutdown();
                break;
            case ActorIdentity:
                // A probe answered by a reachable incarnation — nothing to do.
                break;
            default:
                Unhandled(message);
                break;
        }
    }
    private static bool IsLiveStatus(MemberStatus status) =>
        status is MemberStatus.Joining or MemberStatus.WeaklyUp or MemberStatus.Up or MemberStatus.Leaving;
    private bool IsSelf(Address address) => PeerKey.Of(address) == PeerKey.Of(SelfAddress);
    private void AddKnownPeer(Address address)
    {
        if (!IsSelf(address))
        {
            _knownPeers.Add(PeerKey.Of(address));
        }
    }
    private void TrackMember(Member member)
    {
        var key = PeerKey.Of(member.Address);
        _members[key] = member;
        AddKnownPeer(member.Address);
        if (IsLiveStatus(member.Status) && _probing.Remove(key, out var probe))
        {
            probe.Timer?.Cancel();
        }
    }
    private bool IsLiveMember(Address address, out MemberStatus status)
    {
        if (_members.TryGetValue(PeerKey.Of(address), out var member) && IsLiveStatus(member.Status))
        {
            status = member.Status;
            return true;
        }
        status = MemberStatus.Removed;
        return false;
    }
    /// <summary>Common gate for both signals: ignore after a restart, for self, and for strangers.</summary>
    private bool IsRelevantPeer(Address peer, string what)
    {
        if (_restarting || IsSelf(peer))
        {
            return false;
        }
        var key = PeerKey.Of(peer);
        if (_knownPeers.Contains(key))
        {
            return true;
        }
        if (_ignoredLogged.Add(key))
        {
            Log.Info("[leader-auto-down] {0} {1}, but it is not a seed or member of this cluster; ignored", peer, what);
        }
        return false;
    }
    /// <summary>S1: <paramref name="peer"/> has quarantined this node.</summary>
    private void OnShutOutByPeer(Address peer)
    {
        if (!IsRelevantPeer(peer, $"has quarantined this node ({SelfAddress})"))
        {
            return;
        }
        if (IsLiveMember(peer, out var status))
        {
            Log.Warning(
                "[leader-auto-down] shut out by {0}: it has removed and quarantined this node ({1}) while this node still counts it as a member ({2}), "
                + "so only this side is shut out; tiebreak not needed — this node RESTARTS to rejoin as a fresh incarnation",
                peer, SelfAddress, status);
            Restart();
            return;
        }
        if (_splits.TryGetValue(PeerKey.Of(peer), out var split))
        {
            split.LastEvidenceUtc = UtcNow;
            if (split.Phase == SplitPhase.Winner)
            {
                return;
            }
            // Suspect → the peer has quarantined us too: both sides are shut out, decide now.
            split.Timer.Cancel();
            _splits.Remove(PeerKey.Of(peer));
        }
        Tiebreak(peer, $"shut out by {peer}: it has quarantined this node ({SelfAddress}), and this node has removed it too");
    }
    /// <summary>S2: <paramref name="peer"/> is still running the incarnation this node removed.</summary>
    private void OnPeerStillRunningRemovedIncarnation(Address peer)
    {
        if (!IsRelevantPeer(peer, "refused a handshake with a UID this node has quarantined"))
        {
            return;
        }
        var key = PeerKey.Of(peer);
        if (_splits.TryGetValue(key, out var split))
        {
            split.LastEvidenceUtc = UtcNow;
            return;
        }
        if (IsLiveMember(peer, out _))
        {
            // A live member with a quarantined UID is a transient of Akka's own quarantine path
            // (it is about to be marked terminated and downed); the downing provider handles it.
            return;
        }
        Log.Warning(
            "[leader-auto-down] split from {0}: it is still running the incarnation this node ({1}) removed, so this node has shut it out. "
            + "Giving it {2} to restart itself before the tiebreak (a frozen node that was removed restarts on its own)",
            peer, SelfAddress, _settings.Grace);
        StartSplit(peer, SplitPhase.Suspect);
    }
    private void Tiebreak(Address peer, string why)
    {
        if (Member.AddressOrdering.Compare(SelfAddress, peer) > 0)
        {
            Log.Warning(
                "[leader-auto-down] {0}; both sides are shut out. Tiebreak says RESTART: this node ({1}) has the higher address",
                why, SelfAddress);
            Restart();
            return;
        }
        Log.Warning(
            "[leader-auto-down] {0}; both sides are shut out. Tiebreak says STAY: this node ({1}) has the lower address, so {2} restarts. "
            + "This node restarts too only if {2} has not rejoined and is still split from it after {3}",
            why, SelfAddress, peer, _settings.Grace);
        StartSplit(peer, SplitPhase.Winner);
    }
    private void StartSplit(Address peer, SplitPhase phase)
    {
        _splits[PeerKey.Of(peer)] = new SplitState
        {
            Phase = phase,
            Timer = Scheduler.ScheduleTellOnceCancelable(_settings.Grace, Self, new GraceExpired(peer), Self),
            LastEvidenceUtc = UtcNow,
        };
        // Probe at the short interval during the grace, so "still split" is measured.
        StartProbing(peer, restartBackOff: true);
    }
    private void OnGraceExpired(Address peer)
    {
        if (_restarting || !_splits.Remove(PeerKey.Of(peer), out var split))
        {
            return;
        }
        if (IsLiveMember(peer, out var status))
        {
            Log.Info(
                "[leader-auto-down] {0} rejoined ({1}) within the shut-out grace; this node ({2}) stays",
                peer, status, SelfAddress);
            return;
        }
        var sinceLast = UtcNow - split.LastEvidenceUtc;
        if (sinceLast > _settings.RecentWindow)
        {
            Log.Info(
                "[leader-auto-down] {0} has not rejoined yet, but nothing of its old incarnation has been seen for {1:0}s (it is restarting or down); "
                + "this node ({2}) stays",
                peer, sinceLast.TotalSeconds, SelfAddress);
            return;
        }
        if (split.Phase == SplitPhase.Suspect)
        {
            Tiebreak(peer,
                $"split from {peer}: its old incarnation is still running {_settings.Grace} later and it has not restarted itself");
            return;
        }
        Log.Warning(
            "[leader-auto-down] {0} is still split from this node ({1}) {2} after the tiebreak (last seen {3:0}s ago) and has not rejoined, "
            + "so this node is the stale side after all; this node RESTARTS",
            peer, SelfAddress, _settings.Grace, sinceLast.TotalSeconds);
        Restart();
    }
    private void StartProbing(Address peer, bool restartBackOff = false)
    {
        var key = PeerKey.Of(peer);
        if (_probing.TryGetValue(key, out var existing))
        {
            if (!restartBackOff)
            {
                return;
            }
            existing.Timer?.Cancel();
        }
        var state = existing ?? new ProbeState { Address = peer };
        state.NextInterval = _settings.ProbeInterval;
        state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self);
        _probing[key] = state;
    }
    private void OnProbeTick(Address peer)
    {
        var key = PeerKey.Of(peer);
        if (_restarting || !_probing.TryGetValue(key, out var state))
        {
            return;
        }
        Probe(state.Address);
        // Fixed short interval while a grace runs (to measure "still split"); otherwise back off.
        state.NextInterval = _splits.ContainsKey(key)
            ? _settings.ProbeInterval
            : TimeSpan.FromTicks(Math.Min(state.NextInterval.Ticks * 2, _settings.ProbeMaxInterval.Ticks));
        state.Timer = Scheduler.ScheduleTellOnceCancelable(state.NextInterval, Self, new ProbeTick(peer), Self);
    }
    private void Restart()
    {
        _restarting = true;
        CancelAllTimers();
        _probing.Clear();
        _splits.Clear();
        RestartSelf();
    }
    private void CancelAllTimers()
    {
        foreach (var probe in _probing.Values)
        {
            probe.Timer?.Cancel();
        }
        foreach (var split in _splits.Values)
        {
            split.Timer.Cancel();
        }
    }
 }
@@ -157,6 +157,11 @@ Log.Logger = ZB.MOM.WW.ScadaBridge.Host.ScadaBridgeLogging
 // The catch below uses it to tell that routine stop from a genuine fatal exit (ScadaBridge#76).
 var hostStopping = CancellationToken.None;
 // The Akka hosted service, captured with hostStopping (RunAsync disposes the host's service
 // provider), so the finally block can turn an Akka watchdog stop (self-down / ScadaBridge#132
 // shut-out restart) into a non-zero exit code whichever way RunAsync unwound.
 AkkaHostedService? akkaHostedService = null;
 try
 {
    Log.Information("Starting ScadaBridge host as {Role} on {Hostname}", nodeRole, nodeHostname);
@@ -670,6 +675,7 @@ try
        var app = builder.Build();
        hostStopping = app.Lifetime.ApplicationStopping;
        akkaHostedService = app.Services.GetService<AkkaHostedService>();
        // Apply or validate database migrations (skip when running in test harness)
        if (!string.Equals(configuration["ScadaBridge:Database:SkipMigrations"], "true", StringComparison.OrdinalIgnoreCase))
@@ -864,6 +870,7 @@ try
        var app = builder.Build();
        hostStopping = app.Lifetime.ApplicationStopping;
        akkaHostedService = app.Services.GetService<AkkaHostedService>();
        // Endpoint routing middleware. The gRPC service mapping below and the
        // /metrics scrape endpoint both run on endpoint routing, so UseRouting()
@@ -934,6 +941,8 @@ catch (Exception ex)
 }
 finally
 {
    Program.SetSelfDownExitCode(akkaHostedService);
    await Log.CloseAndFlushAsync();
 }
@@ -955,6 +964,25 @@ public partial class Program
    internal static bool IsGracefulShutdown(Exception ex, CancellationToken hostStopping) =>
        ex is OperationCanceledException && hostStopping.IsCancellationRequested;
    /// <summary>
    /// After <c>RunAsync</c> returns: when the stop was the Akka watchdog's (the ActorSystem
    /// terminated outside host shutdown — a self-down or a ScadaBridge#132 shut-out restart), exit
    /// with the non-zero <see cref="AkkaHostedService.SelfDownExitCode"/> so the service supervisor
    /// restarts the node. Docker <c>restart: unless-stopped</c> restarts on any exit; Windows service
    /// recovery gets the code through the service lifetime (<c>AkkaHostedService.MarkServiceSelfDownExit</c>).
    /// </summary>
    /// <param name="akka">The host's Akka hosted service; null if the host never built.</param>
    internal static void SetSelfDownExitCode(AkkaHostedService? akka)
    {
        if (akka is { TerminatedOutsideHostShutdown: true })
        {
            Environment.ExitCode = AkkaHostedService.SelfDownExitCode;
            Log.Warning(
                "ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code {ExitCode} so the service supervisor restarts it",
                AkkaHostedService.SelfDownExitCode);
        }
    }
    /// <summary>
    /// Extracts the distinct TCP ports from an ASP.NET Core server-URLs string (the
    /// <c>ASPNETCORE_URLS</c> / <c>--urls</c> value). Port-only view of
@@ -327,4 +327,32 @@ public class ClusterOptionsValidatorTests
        Assert.True(result.Failed);
        Assert.Contains("'leader-auto-down'", result.FailureMessage);
    }
    /// <summary>
    /// ScadaBridge#132: a non-positive shut-out grace is refused under leader-auto-down with the
    /// restart on (it would make both sides of a healed partition restart together), and ignored
    /// when the restart is off or another strategy is selected.
    /// </summary>
    /// <param name="strategy">The strategy.</param>
    /// <param name="restart">The shut-out restart switch.</param>
    /// <param name="expectValid">Whether validation must pass.</param>
    [Theory]
    [InlineData("leader-auto-down", true, false)]
    [InlineData("leader-auto-down", false, true)]
    [InlineData("auto-down", true, true)]
    public void LeaderAutoDown_ZeroShutOutGrace(string strategy, bool restart, bool expectValid)
    {
        var options = ValidOptions();
        options.SplitBrainResolverStrategy = strategy;
        options.LeaderAutoDown = new LeaderAutoDownOptions { RestartWhenShutOut = restart, ShutOutGrace = TimeSpan.Zero };
        var result = new ClusterOptionsValidator().Validate(null, options);
        Assert.Equal(expectValid, result.Succeeded);
        if (!expectValid)
        {
            Assert.Contains("ShutOutGrace", result.FailureMessage);
        }
    }
 }
@@ -322,4 +322,46 @@ public class HoconBuilderTests
        Assert.DoesNotContain("leader-auto-down", hocon);
        Assert.DoesNotContain(nameof(LeaderAutoDowningProvider), hocon);
    }
    /// <summary>
    /// ScadaBridge#132: leader-auto-down carries the shut-out restart switch (default ON) and its
    /// grace into the provider's HOCON block, and the provider reads back exactly those values.
    /// </summary>
    /// <param name="enabled">The configured switch.</param>
    [Theory]
    [InlineData(true)]
    [InlineData(false)]
    public void BuildHocon_LeaderAutoDown_EmitsShutOutRestartSettings(bool enabled)
    {
        var cluster = DefaultCluster();
        cluster.SplitBrainResolverStrategy = "leader-auto-down";
        cluster.LeaderAutoDown = new LeaderAutoDownOptions
        {
            RestartWhenShutOut = enabled,
            ShutOutGrace = TimeSpan.FromMilliseconds(45500),
        };
        var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
            DefaultNode(), cluster, new[] { "Central" },
            TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
        var settings = ShutOutRestartSettings.FromConfig(config);
        Assert.Equal(enabled, settings.Enabled);
        Assert.Equal(TimeSpan.FromMilliseconds(45500), settings.Grace);
    }
    /// <summary>ScadaBridge#132: the shut-out restart is ON by default when leader-auto-down is selected.</summary>
    [Fact]
    public void BuildHocon_LeaderAutoDown_ShutOutRestartOnByDefault()
    {
        var cluster = DefaultCluster();
        cluster.SplitBrainResolverStrategy = "leader-auto-down";
        var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
            DefaultNode(), cluster, new[] { "Central" },
            TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
        Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
        Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
    }
 }
@@ -0,0 +1,447 @@
 using System.Collections.Immutable;
 using System.Reflection;
 using Akka.Actor;
 using Akka.Cluster;
 using Akka.Configuration;
 using Akka.Remote;
 using Akka.TestKit.Xunit2;
 using Akka.Util;
 using ZB.MOM.WW.ScadaBridge.Host.Actors;
 namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
 /// <summary>
 /// ScadaBridge#132: the decision logic of <see cref="ShutOutRestartBase"/>. Cluster events and
 /// <see cref="ThisActorSystemQuarantinedEvent"/> are fed straight to the actor; the restart and
 /// probe seams report to the test actor. Covers the single-side shut-out (restart, no tiebreak),
 /// the tiebreak in both orders, the both-sides case where exactly one of two nodes restarts, the
 /// winner's grace, and the probing that makes a healed partition produce the event at all. The
 /// real two-node behaviour is proven in IntegrationTests (<c>ShutOutRestartScenarioTests</c>).
 /// </summary>
 public class ShutOutRestartTests : TestKit
 {
    /// <summary>INFO so the decision-log assertions see every line.</summary>
    public ShutOutRestartTests()
        : base("akka.loglevel = INFO")
    {
    }
    // "a" < "b" < "c" by Akka's member address ordering (host, then port).
    private static readonly Address NodeA = new("akka.tcp", "scadabridge", "a", 8081);
    private static readonly Address NodeB = new("akka.tcp", "scadabridge", "b", 8081);
    private static readonly Address Stranger = new("akka.tcp", "scadabridge", "z", 9999);
    private static readonly ShutOutRestartSettings Fast = new(
        Enabled: true,
        Grace: TimeSpan.FromSeconds(1.5),
        ProbeInterval: TimeSpan.FromMilliseconds(200),
        ProbeMaxInterval: TimeSpan.FromMilliseconds(800));
    private sealed record Restarted(Address Self);
    private sealed record Probed(Address Self, Address Peer);
    /// <summary>Test double: restart and probe report to a probe actor.</summary>
    private sealed class TestShutOutRestart : ShutOutRestartBase
    {
        private readonly IActorRef _probe;
        private readonly Address[] _seeds;
        public TestShutOutRestart(Address self, Address[] seeds, ShutOutRestartSettings settings, IActorRef probe)
            : base(settings)
        {
            SelfAddress = self;
            _seeds = seeds;
            _probe = probe;
        }
        public override Address SelfAddress { get; }
        public override IScheduler Scheduler => Context.System.Scheduler;
        protected override IEnumerable<Address> SeedNodes => _seeds;
        protected override void RestartSelf() => _probe.Tell(new Restarted(SelfAddress));
        protected override void Probe(Address peer) => _probe.Tell(new Probed(SelfAddress, peer));
    }
    private static Member CreateMember(Address address, MemberStatus status)
    {
        var create = typeof(Member).GetMethod(
            "Create",
            BindingFlags.Static | BindingFlags.NonPublic,
            new[] { typeof(UniqueAddress), typeof(int), typeof(MemberStatus), typeof(ImmutableHashSet<string>), typeof(AppVersion) })
            ?? throw new InvalidOperationException("Akka.Cluster.Member.Create(…) not found — Akka internals changed.");
        return (Member)create.Invoke(null, new object[]
        {
            new UniqueAddress(address, 1), 1, status, ImmutableHashSet<string>.Empty, AppVersion.Zero,
        })!;
    }
    private static ClusterEvent.CurrentClusterState StateWith(params Member[] members) =>
        new(
            ImmutableSortedSet.CreateRange(members),
            ImmutableHashSet<Member>.Empty,
            ImmutableHashSet<Address>.Empty,
            null,
            ImmutableDictionary<string, Address>.Empty);
    private IActorRef Node(Address self, Address peer, IActorRef? probe = null, ShutOutRestartSettings? settings = null) =>
        Sys.ActorOf(Props.Create(() => new TestShutOutRestart(
            self, new[] { self, peer }, settings ?? Fast, probe ?? TestActor)));
    /// <summary>Both nodes see each other Up, then each removes the other after downing it (the partition outcome).</summary>
    private static void BothRemovedEachOther(IActorRef node, Address self, Address peer)
    {
        node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up)));
        node.Tell(new ClusterEvent.MemberRemoved(CreateMember(peer, MemberStatus.Removed), MemberStatus.Down));
    }
    private static ThisActorSystemQuarantinedEvent ShutOutBy(Address peer, Address self) => new(self, peer);
    /// <summary>S2 as Akka.Remote 1.5.62 surfaces it: the endpoint writer's InvalidAssociation wrapping the protocol refusal.</summary>
    private static AssociationErrorEvent StillRunningRemoved(Address peer, Address self) =>
        new(
            new Exception(
                $"Association failed with {peer}",
                new AggregateException(new Exception("The remote system has a UID that has been quarantined. Association aborted."))),
            self, peer, inbound: false, Akka.Event.LogLevel.WarningLevel);
    /// <summary>Everything <paramref name="probe"/> receives during <paramref name="window"/>.</summary>
    private static List<object> Drain(Akka.TestKit.TestProbe probe, TimeSpan window)
    {
        var received = new List<object>();
        var until = DateTime.UtcNow + window;
        while (true)
        {
            var left = until - DateTime.UtcNow;
            if (left <= TimeSpan.Zero)
            {
                return received;
            }
            if (probe.TryReceiveOne(out var envelope, left))
            {
                received.Add(envelope.Message);
            }
        }
    }
    /// <summary>
    /// Single side (the frozen node): the peer is still a live member in this node's view, so only
    /// this side can be shut out — it restarts at once, whichever address is higher.
    /// </summary>
    /// <param name="selfIsA">Whether this node is the lower-address node.</param>
    [Theory]
    [InlineData(true)]
    [InlineData(false)]
    public void ShutOut_PeerStillAMember_RestartsWithoutTiebreak(bool selfIsA)
    {
        var (self, peer) = selfIsA ? (NodeA, NodeB) : (NodeB, NodeA);
        var node = Node(self, peer);
        node.Tell(StateWith(CreateMember(self, MemberStatus.Up), CreateMember(peer, MemberStatus.Up)));
        EventFilter.Warning(contains: "tiebreak not needed").ExpectOne(() => node.Tell(ShutOutBy(peer, self)));
        ExpectMsg(new Restarted(self));
        ExpectNoMsg(TimeSpan.FromMilliseconds(300));
    }
    /// <summary>Tiebreak, higher address: both sides removed each other; the higher address restarts at once.</summary>
    [Fact]
    public void ShutOut_PeerRemoved_HigherAddress_Restarts()
    {
        var node = Node(NodeB, NodeA);
        BothRemovedEachOther(node, NodeB, NodeA);
        EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB)));
        ExpectMsg(new Restarted(NodeB));
    }
    /// <summary>
    /// Tiebreak, lower address: it stays; it keeps probing during the grace, and at the end of the
    /// grace — the peer silent, i.e. it restarted — it still stays.
    /// </summary>
    [Fact]
    public void ShutOut_PeerRemoved_LowerAddress_Stays_AndStaysAfterGraceWhenPeerWentQuiet()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeA, NodeB, probe.Ref);
        BothRemovedEachOther(node, NodeA, NodeB);
        EventFilter.Warning(contains: "Tiebreak says STAY").ExpectOne(() => node.Tell(ShutOutBy(NodeB, NodeA)));
        EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { });
        var received = Drain(probe, TimeSpan.FromMilliseconds(500));
        Assert.Contains(new Probed(NodeA, NodeB), received);      // it kept probing its peer
        Assert.DoesNotContain(new Restarted(NodeA), received);    // and never restarted
    }
    /// <summary>
    /// Tiebreak, lower address, but the peer keeps refusing for the whole grace and never rejoins:
    /// the lower node was the stale side after all (single-sided with the lower address) and restarts.
    /// </summary>
    [Fact]
    public async Task ShutOut_PeerRemoved_LowerAddress_StillRefusedAfterGrace_Restarts()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeA, NodeB, probe.Ref);
        BothRemovedEachOther(node, NodeA, NodeB);
        node.Tell(ShutOutBy(NodeB, NodeA));
        var until = DateTime.UtcNow + Fast.Grace + TimeSpan.FromSeconds(1);
        var restarted = false;
        while (DateTime.UtcNow < until && !restarted)
        {
            node.Tell(ShutOutBy(NodeB, NodeA)); // every probe is refused
            await Task.Delay(100);
            while (probe.HasMessages)
            {
                restarted |= probe.ReceiveOne(TimeSpan.Zero) is Restarted;
            }
        }
        Assert.True(restarted, "the lower node never restarted although its peer kept refusing it past the grace");
    }
    /// <summary>Tiebreak, lower address: the peer rejoins during the grace; the lower node stays.</summary>
    [Fact]
    public void ShutOut_PeerRemoved_LowerAddress_PeerRejoins_Stays()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeA, NodeB, probe.Ref);
        BothRemovedEachOther(node, NodeA, NodeB);
        node.Tell(ShutOutBy(NodeB, NodeA));
        node.Tell(ShutOutBy(NodeB, NodeA)); // refused right up to the restart of the peer
        node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeB, MemberStatus.Up)));
        EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { });
        Assert.DoesNotContain(new Restarted(NodeA), Drain(probe, TimeSpan.FromMilliseconds(500)));
    }
    /// <summary>
    /// Both sides shut out (a healed partition): two nodes, each having removed the other, both
    /// receive the event — exactly ONE restarts, and it is the higher address. Run in both
    /// delivery orders so the outcome cannot depend on who hears first.
    /// </summary>
    /// <param name="higherHearsFirst">Whether the higher-address node receives its event first.</param>
    [Theory]
    [InlineData(true)]
    [InlineData(false)]
    public void BothSidesShutOut_ExactlyOneRestarts_TheHigherAddress(bool higherHearsFirst)
    {
        var probe = CreateTestProbe();
        var a = Node(NodeA, NodeB, probe.Ref);
        var b = Node(NodeB, NodeA, probe.Ref);
        BothRemovedEachOther(a, NodeA, NodeB);
        BothRemovedEachOther(b, NodeB, NodeA);
        if (higherHearsFirst)
        {
            b.Tell(ShutOutBy(NodeA, NodeB));
            a.Tell(ShutOutBy(NodeB, NodeA));
        }
        else
        {
            a.Tell(ShutOutBy(NodeB, NodeA));
            b.Tell(ShutOutBy(NodeA, NodeB));
        }
        // Within the grace, the only restart is B's.
        var restarts = Drain(probe, Fast.Grace - TimeSpan.FromMilliseconds(300)).OfType<Restarted>().ToList();
        Assert.Equal(new[] { new Restarted(NodeB) }, restarts);
    }
    /// <summary>The same event delivered many times restarts once.</summary>
    [Fact]
    public void RepeatedShutOut_RestartsOnce()
    {
        var node = Node(NodeB, NodeA);
        node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up)));
        for (var i = 0; i < 5; i++)
        {
            node.Tell(ShutOutBy(NodeA, NodeB));
        }
        ExpectMsg(new Restarted(NodeB));
        ExpectNoMsg(TimeSpan.FromMilliseconds(500));
    }
    /// <summary>A quarantine by an address that is neither a seed nor a member is not this pair's business.</summary>
    [Fact]
    public void ShutOutByStranger_IsIgnored()
    {
        var node = Node(NodeB, NodeA);
        node.Tell(StateWith(CreateMember(NodeB, MemberStatus.Up), CreateMember(NodeA, MemberStatus.Up)));
        EventFilter.Info(contains: "not a seed or member of this cluster").ExpectOne(() => node.Tell(ShutOutBy(Stranger, NodeB)));
        ExpectNoMsg(TimeSpan.FromMilliseconds(500));
    }
    /// <summary>
    /// A peer removed after being DOWNED is probed with a back-off until it is a member again; a
    /// peer that LEFT (Exiting → Removed) is not probed.
    /// </summary>
    [Fact]
    public void DownedPeer_IsProbedUntilItRejoins_LeftPeerIsNot()
    {
        var node = Node(NodeA, NodeB);
        node.Tell(StateWith(CreateMember(NodeA, MemberStatus.Up), CreateMember(NodeB, MemberStatus.Up)));
        node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Exiting));
        ExpectNoMsg(TimeSpan.FromMilliseconds(600));
        node.Tell(new ClusterEvent.MemberRemoved(CreateMember(NodeB, MemberStatus.Removed), MemberStatus.Down));
        ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1));
        ExpectMsg(new Probed(NodeA, NodeB), TimeSpan.FromSeconds(1));
        node.Tell(new ClusterEvent.MemberJoined(CreateMember(NodeB, MemberStatus.Joining)));
        ReceiveWhile<object>(TimeSpan.FromMilliseconds(100), m => m); // a tick already in flight
        ExpectNoMsg(TimeSpan.FromSeconds(1.5));
    }
    /// <summary>
    /// S2 on the higher node (it removed a peer whose old incarnation is still alive): no restart
    /// during the grace — a frozen peer gets the chance to restart itself — then, still split, the
    /// tiebreak restarts it.
    /// </summary>
    [Fact]
    public async Task StillRunningRemoved_HigherAddress_WaitsGrace_ThenRestartsIfStillSplit()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeB, NodeA, probe.Ref);
        BothRemovedEachOther(node, NodeB, NodeA);
        EventFilter.Warning(contains: "Giving it").ExpectOne(() => node.Tell(StillRunningRemoved(NodeA, NodeB)));
        var started = DateTime.UtcNow;
        var restartedAt = (DateTime?)null;
        while (DateTime.UtcNow - started < Fast.Grace + TimeSpan.FromSeconds(1.5) && restartedAt is null)
        {
            node.Tell(StillRunningRemoved(NodeA, NodeB)); // every probe still meets the old incarnation
            await Task.Delay(100);
            while (probe.HasMessages)
            {
                if (probe.ReceiveOne(TimeSpan.Zero) is Restarted)
                {
                    restartedAt = DateTime.UtcNow;
                }
            }
        }
        Assert.NotNull(restartedAt);
        Assert.True(restartedAt.Value - started >= Fast.Grace - TimeSpan.FromMilliseconds(200),
            $"restarted after {(restartedAt.Value - started).TotalSeconds:0.0}s, before the {Fast.Grace} grace");
    }
    /// <summary>
    /// The freeze survivor: S2 (the frozen node's old incarnation is alive), then the frozen node
    /// restarts itself and its old incarnation goes quiet — the survivor does NOT restart, even
    /// though it has the higher address.
    /// </summary>
    [Fact]
    public void StillRunningRemoved_PeerGoesQuiet_NoRestartEvenWhenHigher()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeB, NodeA, probe.Ref);
        BothRemovedEachOther(node, NodeB, NodeA);
        node.Tell(StillRunningRemoved(NodeA, NodeB));
        EventFilter.Info(contains: "nothing of its old incarnation has been seen").ExpectOne(TimeSpan.FromSeconds(4), () => { });
        Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500)));
    }
    /// <summary>S2, then the peer's new incarnation joins during the grace: no restart.</summary>
    [Fact]
    public void StillRunningRemoved_PeerRejoins_NoRestart()
    {
        var probe = CreateTestProbe();
        var node = Node(NodeB, NodeA, probe.Ref);
        BothRemovedEachOther(node, NodeB, NodeA);
        node.Tell(StillRunningRemoved(NodeA, NodeB));
        node.Tell(new ClusterEvent.MemberUp(CreateMember(NodeA, MemberStatus.Up)));
        node.Tell(StillRunningRemoved(NodeA, NodeB)); // stale evidence from before the restart
        EventFilter.Info(contains: "rejoined").ExpectOne(TimeSpan.FromSeconds(4), () => { });
        Assert.DoesNotContain(new Restarted(NodeB), Drain(probe, TimeSpan.FromMilliseconds(500)));
    }
    /// <summary>
    /// A healed partition as Akka 1.5.62 really reports it — S2 on BOTH sides, no S1 — ends with
    /// exactly one restart, the higher address, after the grace; the lower stays.
    /// </summary>
    [Fact]
    public async Task BothSidesStillRunningRemoved_ExactlyOneRestarts_TheHigherAddress()
    {
        var probe = CreateTestProbe();
        var a = Node(NodeA, NodeB, probe.Ref);
        var b = Node(NodeB, NodeA, probe.Ref);
        BothRemovedEachOther(a, NodeA, NodeB);
        BothRemovedEachOther(b, NodeB, NodeA);
        var restarts = new List<Restarted>();
        var until = DateTime.UtcNow + Fast.Grace + Fast.Grace - TimeSpan.FromMilliseconds(400);
        while (DateTime.UtcNow < until)
        {
            if (!restarts.Contains(new Restarted(NodeB)))
            {
                // Until B's old incarnation is gone, each side's probes meet the other's.
                a.Tell(StillRunningRemoved(NodeB, NodeA));
                b.Tell(StillRunningRemoved(NodeA, NodeB));
            }
            await Task.Delay(100);
            while (probe.HasMessages)
            {
                if (probe.ReceiveOne(TimeSpan.Zero) is Restarted r)
                {
                    restarts.Add(r);
                }
            }
        }
        Assert.Equal(new[] { new Restarted(NodeB) }, restarts);
    }
    /// <summary>S1 arriving while an S2 grace runs: the peer has quarantined us too, so the tiebreak runs at once.</summary>
    [Fact]
    public void StillRunningRemoved_ThenShutOut_TiebreaksImmediately()
    {
        var node = Node(NodeB, NodeA);
        BothRemovedEachOther(node, NodeB, NodeA);
        node.Tell(StillRunningRemoved(NodeA, NodeB));
        EventFilter.Warning(contains: "Tiebreak says RESTART").ExpectOne(() => node.Tell(ShutOutBy(NodeA, NodeB)));
        FishForMessage(m => m is Restarted, TimeSpan.FromSeconds(1));
    }
    /// <summary>S2 detection matches Akka 1.5.62's two wordings anywhere in the cause chain, and nothing else.</summary>
    [Fact]
    public void IsQuarantinedUidRefusal_MatchesAkkaWordingOnly()
    {
        Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal(
            new Exception("Association failed with x", new AggregateException(
                new Exception("The remote system has a UID that has been quarantined. Association aborted.")))));
        Assert.True(ShutOutRestartBase.IsQuarantinedUidRefusal(
            new Exception("The remote system [akka.tcp://scadabridge@b:8081] has a UID [42] that has been quarantined. Association aborted.")));
        Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(
            new Exception("Association failed with x", new Exception("Connection refused"))));
        Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(
            new Exception("The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.")));
        Assert.False(ShutOutRestartBase.IsQuarantinedUidRefusal(null));
    }
    /// <summary>Settings: defaults when keys are missing; the switch and durations are read when present.</summary>
    [Fact]
    public void Settings_FromConfig_DefaultsAndOverrides()
    {
        var defaults = ShutOutRestartSettings.FromConfig(ConfigurationFactory.Empty);
        Assert.Equal(new ShutOutRestartSettings(true, TimeSpan.FromSeconds(60), TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60)), defaults);
        var set = ShutOutRestartSettings.FromConfig(ConfigurationFactory.ParseString($@"
 {ShutOutRestartSettings.EnabledPath} = off
 {ShutOutRestartSettings.GracePath} = 2500ms
 {ShutOutRestartSettings.ProbeIntervalPath} = 1s
 {ShutOutRestartSettings.ProbeMaxIntervalPath} = 4s"));
        Assert.Equal(new ShutOutRestartSettings(false, TimeSpan.FromMilliseconds(2500), TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(4)), set);
    }
 }
@@ -69,6 +69,8 @@ public sealed class UnexpectedTerminationTests
        await Task.Delay(500);
        lifetime.Received(1).StopApplication();
        // ScadaBridge#132: Program turns this flag into the non-zero SelfDownExitCode.
        Assert.True(service.TerminatedOutsideHostShutdown);
    }
    /// <summary>Verifies that a normal stop does not trigger an application stop.</summary>
@@ -84,5 +86,19 @@ public sealed class UnexpectedTerminationTests
        await Task.Delay(500);
        lifetime.DidNotReceive().StopApplication();
        Assert.False(service.TerminatedOutsideHostShutdown);
    }
    /// <summary>
    /// ScadaBridge#132: the self-down exit code is non-zero (Windows service recovery with
    /// failureflag 1 restarts only on a non-zero exit) and is only pushed into a Windows service
    /// lifetime — a console / docker lifetime is left alone.
    /// </summary>
    [Fact]
    public void SelfDownExitCode_IsNonZero_AndOnlyAppliedToAServiceLifetime()
    {
        Assert.NotEqual(0, AkkaHostedService.SelfDownExitCode);
        Assert.False(AkkaHostedService.MarkServiceSelfDownExit(null));
        Assert.False(AkkaHostedService.MarkServiceSelfDownExit(Substitute.For<IHostLifetime>()));
    }
 }
@@ -0,0 +1,204 @@
 using Akka.Actor;
 using Akka.Cluster;
 using Akka.Configuration;
 using Akka.Remote;
 using Akka.Remote.Transport;
 using Xunit;
 using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
 using ZB.MOM.WW.ScadaBridge.Host;
 using ZB.MOM.WW.ScadaBridge.Host.Actors;
 namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
 /// <summary>
 /// ScadaBridge#132, on real in-process two-node clusters built from the production
 /// <c>BuildHocon</c> (same harness as <see cref="LeaderAutoDownScenarioTests"/>, with Akka's
 /// throttler transport to cut and restore the network without killing either node):
 /// <list type="bullet">
 /// <item>the FROZEN-node shape — only one side downed the other before the network came back —
 /// ends with the shut-out node restarting (its <c>ActorSystem</c> terminates) and the other
 /// staying Up;</item>
 /// <item>a partition held past the window and then HEALED — each side downed the other — ends
 /// with exactly one restart, the higher address; a fresh incarnation on that address rejoins the
 /// survivor as a 2-member cluster, and the survivor is still Up after its grace;</item>
 /// <item>with the switch OFF, the healed partition keeps both sides up (the #116 gate-4 contract
 /// before #132) — the positive control that the restart comes from the feature.</item>
 /// </list>
 /// "Restart" in-process is the termination of the node's <c>ActorSystem</c>; the Host turns that
 /// into a process exit (covered by <c>UnexpectedTerminationTests</c>).
 /// </summary>
 public sealed class ShutOutRestartScenarioTests : IAsyncLifetime
 {
    private static readonly TimeSpan Heartbeat = TimeSpan.FromMilliseconds(500);
    private static readonly TimeSpan Threshold = TimeSpan.FromSeconds(2);
    private readonly List<ActorSystem> _systems = new();
    private ActorSystem StartNode(
        int selfPort, int peerPort, TimeSpan stableAfter, bool restartWhenShutOut = true, TimeSpan? grace = null)
    {
        const string scheme = "akka.trttl.tcp";
        var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
        var clusterOptions = new ClusterOptions
        {
            SeedNodes = new List<string>
            {
                $"{scheme}://scadabridge@127.0.0.1:{selfPort}",
                $"{scheme}://scadabridge@127.0.0.1:{peerPort}",
            },
            SplitBrainResolverStrategy = "leader-auto-down",
            StableAfter = stableAfter,
            HeartbeatInterval = Heartbeat,
            FailureDetectionThreshold = Threshold,
            MinNrOfMembers = 1,
            LeaderAutoDown = new LeaderAutoDownOptions
            {
                RestartWhenShutOut = restartWhenShutOut,
                ShutOutGrace = grace ?? TimeSpan.FromSeconds(20),
            },
        };
        var hocon = AkkaHostedService.BuildHocon(
            nodeOptions, clusterOptions, new[] { "Central" },
            TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(60));
        var overrides = "akka.coordinated-shutdown.run-by-actor-system-terminate = off\n"
            + "akka.remote.dot-netty.tcp.applied-adapters = [trttl]\n"
            + "akka.remote.retry-gate-closed-for = 500ms\n"
            // Fast probes so a healed partition is noticed within the test's patience.
            + "scadabridge.cluster.leader-auto-down.shut-out-probe-interval = 1s\n"
            + "scadabridge.cluster.leader-auto-down.shut-out-probe-max-interval = 2s\n";
        var config = ConfigurationFactory.ParseString(overrides)
            .WithFallback(ConfigurationFactory.ParseString(hocon));
        var system = ActorSystem.Create("scadabridge", config);
        _systems.Add(system);
        return system;
    }
    private static Task<bool> Throttle(ActorSystem from, ActorSystem to, ThrottleMode mode) =>
        ((RemoteActorRefProvider)((ExtendedActorSystem)from).Provider).Transport.ManagementCommand(
            new SetThrottle(Akka.Cluster.Cluster.Get(to).SelfAddress, ThrottleTransportAdapter.Direction.Both, mode));
    private async Task<(ActorSystem Lower, ActorSystem Higher, int LowerPort, int HigherPort)> StartPair(
        TimeSpan stableLower, TimeSpan stableHigher, bool restartWhenShutOut = true)
    {
        var p1 = TwoNodeClusterFixture.GetFreeTcpPort();
        var p2 = TwoNodeClusterFixture.GetFreeTcpPort();
        var (lowerPort, higherPort) = p1 < p2 ? (p1, p2) : (p2, p1);
        var lower = StartNode(lowerPort, higherPort, stableLower, restartWhenShutOut);
        await TwoNodeClusterFixture.WaitForMembersUp(lower, 1, TimeSpan.FromSeconds(30));
        var higher = StartNode(higherPort, lowerPort, stableHigher, restartWhenShutOut);
        await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(30));
        await TwoNodeClusterFixture.WaitForMembersUp(higher, 2, TimeSpan.FromSeconds(30));
        return (lower, higher, lowerPort, higherPort);
    }
    private static async Task<bool> Terminates(ActorSystem system, TimeSpan within)
    {
        var done = await Task.WhenAny(system.WhenTerminated, Task.Delay(within));
        return done == system.WhenTerminated;
    }
    /// <summary>
    /// Frozen-node shape: only the LOWER node downs the higher one (the higher one's window is far
    /// longer than the cut), then the network returns. The higher node — still counting the lower as
    /// a member — is refused, restarts without a tiebreak; the lower node stays Up. Then the same with
    /// roles swapped, so the single-side rule is proven independent of the address order.
    /// </summary>
    /// <param name="shutOutIsHigher">Whether the shut-out (frozen) node has the higher address.</param>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Theory]
    [InlineData(true)]
    [InlineData(false)]
    public async Task SingleSideShutOut_ShutOutNodeRestarts_OtherStaysUp(bool shutOutIsHigher)
    {
        var quick = TimeSpan.FromSeconds(2);
        var never = TimeSpan.FromMinutes(5);
        var (lower, higher, _, _) = shutOutIsHigher
            ? await StartPair(quick, never)
            : await StartPair(never, quick);
        var (survivor, shutOut) = shutOutIsHigher ? (lower, higher) : (higher, lower);
        Assert.True(await Throttle(lower, higher, Blackhole.Instance));
        await TwoNodeClusterFixture.WaitForMemberRemoved(survivor, Akka.Cluster.Cluster.Get(shutOut).SelfAddress, TimeSpan.FromSeconds(30));
        Assert.Contains(Akka.Cluster.Cluster.Get(shutOut).State.Members,
            m => m.Address == Akka.Cluster.Cluster.Get(survivor).SelfAddress); // it still counts the survivor
        Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
        Assert.True(await Terminates(shutOut, TimeSpan.FromSeconds(30)),
            "the shut-out node kept running as a second active node");
        Assert.False(await Terminates(survivor, TimeSpan.FromSeconds(5)), "the survivor restarted too");
        Assert.Equal(MemberStatus.Up, Akka.Cluster.Cluster.Get(survivor).SelfMember.Status);
    }
    /// <summary>
    /// A partition past the window, then healed: both sides downed and removed each other, so both
    /// are refused — exactly ONE restarts, the higher address. A fresh incarnation on the higher
    /// port (what the service supervisor starts) rejoins the survivor, and the survivor is still Up
    /// after its grace has run out.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task HealedPartition_BothShutOut_OnlyTheHigherAddressRestarts_AndRejoins()
    {
        var stable = TimeSpan.FromSeconds(2);
        var (lower, higher, lowerPort, higherPort) = await StartPair(stable, stable);
        var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress;
        var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress;
        Assert.True(await Throttle(lower, higher, Blackhole.Instance));
        await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30));
        await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30));
        Assert.False(lower.WhenTerminated.IsCompleted || higher.WhenTerminated.IsCompleted,
            "a node restarted while the partition was still in place");
        Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
        Assert.True(await Terminates(higher, TimeSpan.FromSeconds(30)),
            "the higher-address node did not restart after the partition healed");
        Assert.False(lower.WhenTerminated.IsCompleted, "the lower-address node restarted as well");
        // The supervisor's restart: a fresh incarnation on the same address rejoins the survivor.
        var reborn = StartNode(higherPort, lowerPort, stable);
        await TwoNodeClusterFixture.WaitForMembersUp(lower, 2, TimeSpan.FromSeconds(60));
        await TwoNodeClusterFixture.WaitForMembersUp(reborn, 2, TimeSpan.FromSeconds(60));
        // Past the survivor's 20s grace: it saw the peer rejoin, so it stays.
        Assert.False(await Terminates(lower, TimeSpan.FromSeconds(25)), "the survivor restarted after its peer rejoined");
        Assert.Equal(2, Akka.Cluster.Cluster.Get(lower).State.Members.Count(m => m.Status == MemberStatus.Up));
        Assert.Empty(Akka.Cluster.Cluster.Get(lower).State.Unreachable);
    }
    /// <summary>
    /// Switch OFF: the same healed partition keeps BOTH sides running (the pre-#132 dual-active)
    /// — the positive control that the restarts above come from the feature.
    /// </summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    [Fact]
    public async Task HealedPartition_SwitchOff_BothStayUp()
    {
        var stable = TimeSpan.FromSeconds(2);
        var (lower, higher, _, _) = await StartPair(stable, stable, restartWhenShutOut: false);
        var lowerAddress = Akka.Cluster.Cluster.Get(lower).SelfAddress;
        var higherAddress = Akka.Cluster.Cluster.Get(higher).SelfAddress;
        Assert.True(await Throttle(lower, higher, Blackhole.Instance));
        await TwoNodeClusterFixture.WaitForMemberRemoved(lower, higherAddress, TimeSpan.FromSeconds(30));
        await TwoNodeClusterFixture.WaitForMemberRemoved(higher, lowerAddress, TimeSpan.FromSeconds(30));
        Assert.True(await Throttle(lower, higher, Unthrottled.Instance));
        Assert.False(await Terminates(higher, TimeSpan.FromSeconds(15)), "a node restarted with the switch off");
        Assert.False(lower.WhenTerminated.IsCompleted, "a node restarted with the switch off");
    }
    /// <summary>Prepares the fixture before the tests run.</summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    public Task InitializeAsync() => Task.CompletedTask;
    /// <summary>Releases the resources held by this fixture asynchronously.</summary>
    /// <returns>A task that represents the asynchronous operation.</returns>
    public async Task DisposeAsync()
    {
        foreach (var s in _systems)
        {
            try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
        }
    }
 }