diff --git a/CLAUDE.md b/CLAUDE.md index dbae4824..e17bfd4b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -199,7 +199,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*. ### Cluster & Failover -- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it. +- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix. - Both nodes are seed nodes. `min-nr-of-members = 1`. - Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`). - CoordinatedShutdown for graceful singleton handover. diff --git a/docs/plans/2026-11-09-autodowning-replacement-plan.md b/docs/plans/2026-11-09-autodowning-replacement-plan.md index 26caf234..c93ed01a 100644 --- a/docs/plans/2026-11-09-autodowning-replacement-plan.md +++ b/docs/plans/2026-11-09-autodowning-replacement-plan.md @@ -8,6 +8,16 @@ `DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value. Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the docker failover drill (section 6) on the new value, then the per-environment rollout in section 5. + +**Update 2026-11-10 (ScadaBridge#106 complete):** every downing decision is now logged with a `[leader-auto-down]` +prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed, +timed out on a non-leader → pending, leadership gained/lost, and `DOWNING` at Warning before `Cluster.Down`). New +real-cluster scenarios in `IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs`: joint cold start, lone cold start +with the peer dead then the peer joining, a flapping peer (Akka `trttl` throttler blackhole, unreachable shorter than the +window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held +past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (`auto-down` → +`AutoDowning`, `keep-oldest` → `SplitBrainResolverProvider`). The docker drill and rollout are tracked as the deploy gate +**ScadaBridge#116**. Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md). ## 1. Current state (read from the code) diff --git a/docs/requirements/Component-ClusterInfrastructure.md b/docs/requirements/Component-ClusterInfrastructure.md index f299c710..207fdf8c 100644 --- a/docs/requirements/Component-ClusterInfrastructure.md +++ b/docs/requirements/Component-ClusterInfrastructure.md @@ -102,7 +102,7 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate ### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default) -Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`. +Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`. ### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21) diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs index 37f4229b..5eb0d3ac 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs @@ -119,7 +119,6 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider internal sealed class LeaderAutoDown : LeaderAutoDownBase { private readonly Akka.Cluster.Cluster _cluster; - private readonly ILoggingAdapter _log = Context.GetLogger(); /// Props for the downing actor. /// How long a member must stay unreachable before it is downed. @@ -165,7 +164,6 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase throw new InvalidOperationException("Must be leader to down node"); } - _log.Info("Leader is auto-downing unreachable node [{0}]", node); _cluster.Down(node); } } @@ -174,6 +172,15 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase /// The cluster-free decision logic of — port of Akka's internal /// AutoDownBase, split out (as Akka does) so the leader / reachability semantics can be /// unit tested by feeding cluster events directly. +/// +/// Every downing decision is logged (ScadaBridge#106) — with the [leader-auto-down] +/// prefix so an operator can grep one node's log for the whole story: a peer observed +/// unreachable (and the window it now has), a peer skipped because it is already Down/Exiting, +/// a pending timer cancelled because the peer became reachable again or was removed, a timer +/// that fired on a non-leader (kept pending), leadership gained/lost, and the actual Down +/// (logged at Warning, before is called). Akka's own AutoDown logs +/// only the last of these. +/// /// internal abstract class LeaderAutoDownBase : UntypedActor { @@ -193,6 +200,9 @@ internal abstract class LeaderAutoDownBase : UntypedActor ImmutableDictionary.Empty; private ImmutableHashSet _pendingUnreachable = ImmutableHashSet.Empty; + /// Decision log; every downing decision is written here. + private readonly ILoggingAdapter _log = Context.GetLogger(); + /// Creates the base logic. /// How long a member must stay unreachable before it is downed. protected LeaderAutoDownBase(TimeSpan unreachableAfter) @@ -229,6 +239,10 @@ internal abstract class LeaderAutoDownBase : UntypedActor { case ClusterEvent.CurrentClusterState state: IsLeader = state.Leader != null && state.Leader.Equals(SelfAddress); + _log.Info( + "[leader-auto-down] started on {0}: leader={1} (this node is leader: {2}), unreachable-after={3}, already unreachable=[{4}]", + SelfAddress, state.Leader?.ToString() ?? "none", IsLeader, _unreachableAfter, + string.Join(", ", state.Unreachable.Select(m => m.Address))); foreach (var member in state.Unreachable) { UnreachableMember(member); @@ -240,19 +254,31 @@ internal abstract class LeaderAutoDownBase : UntypedActor break; case ClusterEvent.ReachableMember reachable: - Remove(reachable.Member.UniqueAddress); + Remove(reachable.Member.UniqueAddress, "became reachable again"); break; case ClusterEvent.MemberRemoved removed: - Remove(removed.Member.UniqueAddress); + Remove(removed.Member.UniqueAddress, "was removed from the cluster"); break; case ClusterEvent.LeaderChanged leaderChanged: + var wasLeader = IsLeader; IsLeader = leaderChanged.Leader != null && leaderChanged.Leader.Equals(SelfAddress); + if (wasLeader != IsLeader) + { + _log.Info( + "[leader-auto-down] {0} {1} leadership (leader is now {2}); pending unreachable=[{3}]", + SelfAddress, IsLeader ? "gained" : "lost", leaderChanged.Leader?.ToString() ?? "none", + string.Join(", ", _pendingUnreachable.Select(n => n.Address))); + } + if (IsLeader) { foreach (var node in _pendingUnreachable) { + _log.Warning( + "[leader-auto-down] DOWNING {0}: it timed out as unreachable while this node was not leader, and this node ({1}) is now leader", + node.Address, SelfAddress); Down(node.Address); } @@ -274,7 +300,15 @@ internal abstract class LeaderAutoDownBase : UntypedActor private void UnreachableMember(Member member) { - if (!SkipMemberStatus.Contains(member.Status) && !_scheduledUnreachable.ContainsKey(member.UniqueAddress)) + if (SkipMemberStatus.Contains(member.Status)) + { + _log.Info( + "[leader-auto-down] {0} is unreachable but already {1}; not scheduling a Down", + member.Address, member.Status); + return; + } + + if (!_scheduledUnreachable.ContainsKey(member.UniqueAddress)) { ScheduleUnreachable(member.UniqueAddress); } @@ -288,6 +322,10 @@ internal abstract class LeaderAutoDownBase : UntypedActor return; } + _log.Warning( + "[leader-auto-down] {0} is unreachable; it will be downed if still unreachable after {1} (this node {2} is leader: {3})", + node.Address, _unreachableAfter, SelfAddress, IsLeader); + var cancelable = Scheduler.ScheduleTellOnceCancelable( _unreachableAfter, Self, new UnreachableTimeout(node), Self); _scheduledUnreachable = _scheduledUnreachable.Add(node, cancelable); @@ -297,19 +335,31 @@ internal abstract class LeaderAutoDownBase : UntypedActor { if (IsLeader) { + _log.Warning( + "[leader-auto-down] DOWNING {0}: unreachable for {1} and this node ({2}) is leader", + node.Address, _unreachableAfter, SelfAddress); Down(node.Address); } else { + _log.Info( + "[leader-auto-down] {0} unreachable for {1} but this node ({2}) is not leader; kept pending until it becomes leader", + node.Address, _unreachableAfter, SelfAddress); _pendingUnreachable = _pendingUnreachable.Add(node); } } - private void Remove(UniqueAddress node) + private void Remove(UniqueAddress node, string reason) { - if (_scheduledUnreachable.TryGetValue(node, out var cancelable)) + var wasScheduled = _scheduledUnreachable.TryGetValue(node, out var cancelable); + if (wasScheduled) { - cancelable.Cancel(); + cancelable!.Cancel(); + } + + if (wasScheduled || _pendingUnreachable.Contains(node)) + { + _log.Info("[leader-auto-down] {0} {1}; its pending Down is cancelled", node.Address, reason); } _scheduledUnreachable = _scheduledUnreachable.Remove(node); diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs index ff408735..368b5481 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs @@ -18,6 +18,12 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests; /// public class LeaderAutoDownTests : TestKit { + /// INFO so the decision-log assertions can see the Info-level decisions. + public LeaderAutoDownTests() + : base("akka.loglevel = INFO") + { + } + private static readonly Address MemberA = new("akka.tcp", "sys", "a", 2552); private static readonly Address MemberB = new("akka.tcp", "sys", "b", 2552); private static readonly Address MemberC = new("akka.tcp", "sys", "c", 2552); @@ -212,4 +218,107 @@ public class LeaderAutoDownTests : TestKit ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3)); ExpectNoMsg(TimeSpan.FromSeconds(2)); } + + /// + /// A flapping member — unreachable, reachable again inside the window, repeatedly, for far + /// longer than one window in total — is never downed. + /// + [Fact] + public void FlappingMember_ReachableAgainInsideEachWindow_IsNeverDowned() + { + var a = AutoDownActor(TimeSpan.FromMilliseconds(1500)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + for (var i = 0; i < 5; i++) + { + a.Tell(new ClusterEvent.UnreachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(700)); + a.Tell(new ClusterEvent.ReachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(100)); + } + + ExpectNoMsg(TimeSpan.FromSeconds(2)); + } + + /// + /// When a flapping member finally stays unreachable, the window restarts from its LAST + /// unreachable — an earlier flap's timer must not fire — and it is downed exactly once. + /// + [Fact] + public void FlappingMember_ThatStaysUnreachable_IsDownedOneFullWindowAfterLastUnreachable() + { + var a = AutoDownActor(TimeSpan.FromMilliseconds(1500)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + a.Tell(new ClusterEvent.ReachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(200)); + a.Tell(new ClusterEvent.UnreachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + a.Tell(new ClusterEvent.ReachableMember(b)); + ExpectNoMsg(TimeSpan.FromMilliseconds(200)); + a.Tell(new ClusterEvent.UnreachableMember(b)); // final: stays unreachable from here + + // The first flap's timer would fire ~0.1s from here; the final window ends ~1.5s from here. + ExpectNoMsg(TimeSpan.FromMilliseconds(1100)); + ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(2)); + ExpectNoMsg(TimeSpan.FromSeconds(2)); + } + + /// The Down decision is logged at Warning, naming the member and the window. + [Fact] + public void DecisionLog_Down_IsLoggedAtWarning() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + EventFilter.Warning(contains: $"[leader-auto-down] DOWNING {MemberB}").ExpectOne(() => + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up)))); + ExpectMsg(new DownCalled(MemberB)); + } + + /// Observing an unreachable member (and its window) is logged at Warning. + [Fact] + public void DecisionLog_UnreachableScheduled_IsLoggedAtWarning() + { + var a = AutoDownActor(TimeSpan.FromSeconds(30)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + EventFilter.Warning(contains: $"{MemberB} is unreachable; it will be downed if still unreachable after").ExpectOne(() => + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up)))); + } + + /// A cancelled Down (member reachable again) is logged. + [Fact] + public void DecisionLog_ReachableAgain_CancelIsLogged() + { + var a = AutoDownActor(TimeSpan.FromSeconds(30)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(b)); + EventFilter.Info(contains: $"{MemberB} became reachable again; its pending Down is cancelled").ExpectOne(() => + a.Tell(new ClusterEvent.ReachableMember(b))); + } + + /// A timeout on a non-leader (kept pending) is logged, and so is the later Down on becoming leader. + [Fact] + public void DecisionLog_PendingOnNonLeader_ThenDownOnBecomingLeader_AreLogged() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + EventFilter.Info(contains: "is not leader; kept pending").ExpectOne(() => + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up)))); + EventFilter.Warning(contains: $"DOWNING {MemberC}: it timed out as unreachable while this node was not leader").ExpectOne(() => + a.Tell(new ClusterEvent.LeaderChanged(MemberA))); + ExpectMsg(new DownCalled(MemberC)); + } + + /// Skipping an already Down member is logged. + [Fact] + public void DecisionLog_SkippedDownMember_IsLogged() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + EventFilter.Info(contains: "but already Down; not scheduling a Down").ExpectOne(() => + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Down)))); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs new file mode 100644 index 00000000..a134fecf --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs @@ -0,0 +1,257 @@ +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Akka.Remote; +using Akka.Remote.Transport; +using Xunit; +using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure; +using ZB.MOM.WW.ScadaBridge.Host; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +/// +/// ScadaBridge#106: the operational scenarios of the in-repo +/// beyond the hard-crash clones in — cold starts +/// (together, and alone with the peer dead: the shape that makes SBR down-if-alone +/// self-down on a cold recreate), a FLAPPING peer that keeps going unreachable for less than +/// the window, a real partition (the accepted dual-active trade), and the opt-in switch being +/// off (the old providers are what Akka loads). +/// +/// Nodes are built from the production BuildHocon output with the shipped self-first +/// seed ordering, exactly like . The flapping and +/// partition cases add Akka's own throttler transport adapter (trttl, the one the +/// MultiNodeTestRunner uses for Blackhole) so the network can be cut and restored +/// without killing either node. +/// +/// +public sealed class LeaderAutoDownScenarioTests : IAsyncLifetime +{ + private const string Strategy = "leader-auto-down"; + private static readonly TimeSpan Heartbeat = TimeSpan.FromMilliseconds(500); + private static readonly TimeSpan Threshold = TimeSpan.FromSeconds(2); + + private readonly List _systems = new(); + + private ActorSystem StartNode( + int selfPort, int peerPort, string strategy = Strategy, TimeSpan? stableAfter = null, bool throttled = false) + { + // With the throttler applied the published address scheme becomes akka.trttl.tcp, so the + // seeds must carry it too or the nodes never find each other. + var scheme = throttled ? "akka.trttl.tcp" : "akka.tcp"; + var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort }; + var clusterOptions = new ClusterOptions + { + SeedNodes = new List + { + $"{scheme}://scadabridge@127.0.0.1:{selfPort}", + $"{scheme}://scadabridge@127.0.0.1:{peerPort}", + }, + SplitBrainResolverStrategy = strategy, + StableAfter = stableAfter ?? TimeSpan.FromSeconds(3), + HeartbeatInterval = Heartbeat, + FailureDetectionThreshold = Threshold, + MinNrOfMembers = 1, + DownIfAlone = true, + }; + // Transport failure detection far above the cluster window: a blackhole must make the + // MEMBER unreachable (what the downing provider reacts to), not tear the association down. + var hocon = AkkaHostedService.BuildHocon( + nodeOptions, clusterOptions, new[] { "Central" }, + TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(60)); + var overrides = "akka.coordinated-shutdown.run-by-actor-system-terminate = off\n" + + (throttled ? "akka.remote.dot-netty.tcp.applied-adapters = [trttl]\n" : string.Empty); + var config = ConfigurationFactory.ParseString(overrides) + .WithFallback(ConfigurationFactory.ParseString(hocon)); + var system = ActorSystem.Create("scadabridge", config); + _systems.Add(system); + return system; + } + + private static Task Throttle(ActorSystem from, ActorSystem to, ThrottleMode mode) => + ((RemoteActorRefProvider)((ExtendedActorSystem)from).Provider).Transport.ManagementCommand( + new SetThrottle(Akka.Cluster.Cluster.Get(to).SelfAddress, ThrottleTransportAdapter.Direction.Both, mode)); + + private static void AssertTwoUpNoneUnreachable(ActorSystem node, string because) + { + var cluster = Akka.Cluster.Cluster.Get(node); + Assert.False(cluster.IsTerminated, $"{cluster.SelfAddress}: Cluster extension terminated ({because})"); + Assert.Equal(MemberStatus.Up, cluster.SelfMember.Status); + Assert.Equal(2, cluster.State.Members.Count(m => m.Status == MemberStatus.Up)); + Assert.Empty(cluster.State.Unreachable); + } + + /// + /// Both nodes cold-start together: they converge on ONE two-member cluster and, once + /// formed, nobody is downed — held for well past failure detection plus the window. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task BothNodesColdStartTogether_FormOneCluster_AndNeitherIsDowned() + { + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); + var nodeA = StartNode(portA, portB); + var nodeB = StartNode(portB, portA); + + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60)); + Assert.IsType(Akka.Cluster.Cluster.Get(nodeA).DowningProvider); + + await Task.Delay(Threshold + TimeSpan.FromSeconds(3) + TimeSpan.FromSeconds(5)); + AssertTwoUpNoneUnreachable(nodeA, "after a joint cold start"); + AssertTwoUpNoneUnreachable(nodeB, "after a joint cold start"); + } + + /// + /// A lone cold start with the peer DEAD (the cold-recreate shape where SBR keep-oldest + /// down-if-alone downs the lone node) forms a one-node cluster that stays Up past the + /// window; when the peer is later cold-started it joins, and nobody is downed. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task LoneColdStart_PeerDead_StaysUp_ThenPeerColdStartsAndJoins() + { + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening yet + var nodeA = StartNode(portA, portB); + + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30)); + await Task.Delay(Threshold + TimeSpan.FromSeconds(3) + TimeSpan.FromSeconds(3)); + var clusterA = Akka.Cluster.Cluster.Get(nodeA); + Assert.False(clusterA.IsTerminated, "lone cold-started node downed itself"); + Assert.Equal(MemberStatus.Up, clusterA.SelfMember.Status); + + var nodeB = StartNode(portB, portA); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60)); + await Task.Delay(TimeSpan.FromSeconds(5)); + AssertTwoUpNoneUnreachable(nodeA, "after the peer joined"); + AssertTwoUpNoneUnreachable(nodeB, "after the peer joined"); + } + + /// + /// A FLAPPING peer — cut off long enough to be marked unreachable, restored before the + /// window ends, over and over, for far longer than the window in total — is never downed: + /// each return to reachable cancels its pending Down and the next cut starts a fresh window. + /// The test first proves the peer really did go unreachable on (almost) every cut, so the + /// "never downed" outcome is not vacuous. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task FlappingPeer_UnreachableShorterThanWindow_IsNeverDowned() + { + var stableAfter = TimeSpan.FromSeconds(8); + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); + var nodeA = StartNode(portA, portB, stableAfter: stableAfter, throttled: true); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30)); + var nodeB = StartNode(portB, portA, stableAfter: stableAfter, throttled: true); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(30)); + + var clusterA = Akka.Cluster.Cluster.Get(nodeA); + const int flaps = 4; + var flapsSeenUnreachable = 0; + var started = DateTime.UtcNow; + for (var i = 0; i < flaps; i++) + { + Assert.True(await Throttle(nodeA, nodeB, Blackhole.Instance), "SetThrottle(Blackhole) rejected"); + var seen = false; + var cutUntil = DateTime.UtcNow + TimeSpan.FromSeconds(4.5); + while (DateTime.UtcNow < cutUntil) + { + seen |= clusterA.State.Unreachable.Any(); + await Task.Delay(100); + } + + Assert.True(await Throttle(nodeA, nodeB, Unthrottled.Instance), "SetThrottle(Unthrottled) rejected"); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(15)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(15)); + if (seen) + { + flapsSeenUnreachable++; + } + } + + // Positive control: the cuts really produced unreachability, and the flapping lasted + // longer than one window — a provider that did not cancel on reachable would have downed. + Assert.True(flapsSeenUnreachable >= flaps - 1, + $"only {flapsSeenUnreachable}/{flaps} cuts made the peer unreachable — the flap never happened"); + Assert.True(DateTime.UtcNow - started > stableAfter * 2); + + await Task.Delay(stableAfter + TimeSpan.FromSeconds(2)); + AssertTwoUpNoneUnreachable(nodeA, "flapping peer"); + AssertTwoUpNoneUnreachable(nodeB, "flapping peer"); + } + + /// + /// A real partition longer than the window: each side is leader of its own reachable set, + /// so each downs the other and BOTH stay Up — the dual-active outcome the availability-first + /// decision accepts (identical to AutoDowning). Also the positive control for the flapping + /// test: the same cut, held past the window, does down. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp() + { + var stableAfter = TimeSpan.FromSeconds(3); + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); + var nodeA = StartNode(portA, portB, stableAfter: stableAfter, throttled: true); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30)); + var nodeB = StartNode(portB, portA, stableAfter: stableAfter, throttled: true); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(30)); + var addressA = Akka.Cluster.Cluster.Get(nodeA).SelfAddress; + var addressB = Akka.Cluster.Cluster.Get(nodeB).SelfAddress; + + Assert.True(await Throttle(nodeA, nodeB, Blackhole.Instance)); + + await TwoNodeClusterFixture.WaitForMemberRemoved(nodeA, addressB, TimeSpan.FromSeconds(30)); + await TwoNodeClusterFixture.WaitForMemberRemoved(nodeB, addressA, TimeSpan.FromSeconds(30)); + foreach (var node in new[] { nodeA, nodeB }) + { + var cluster = Akka.Cluster.Cluster.Get(node); + Assert.False(cluster.IsTerminated, $"{cluster.SelfAddress} downed itself during the partition"); + Assert.Equal(MemberStatus.Up, cluster.SelfMember.Status); + Assert.Equal(cluster.SelfAddress, cluster.State.Leader); + } + } + + /// + /// The opt-in switch OFF: the default auto-down still loads Akka's own + /// AutoDowning and keep-oldest still loads the SBR provider — the in-repo + /// provider is loaded only for leader-auto-down. + /// + /// The configured strategy. + /// The downing provider Akka must have instantiated. + /// A task that represents the asynchronous operation. + [Theory] + [InlineData("auto-down", typeof(AutoDowning))] + [InlineData("keep-oldest", typeof(Akka.Cluster.SBR.SplitBrainResolverProvider))] + [InlineData(Strategy, typeof(LeaderAutoDowningProvider))] + public async Task SwitchSelectsProvider_OldPathsUnchangedWhenOff(string strategy, Type expectedProvider) + { + var portA = TwoNodeClusterFixture.GetFreeTcpPort(); + var portB = TwoNodeClusterFixture.GetFreeTcpPort(); + var nodeA = StartNode(portA, portB, strategy: strategy); + await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30)); + + Assert.IsType(expectedProvider, Akka.Cluster.Cluster.Get(nodeA).DowningProvider); + } + + /// Prepares the fixture before the tests run. + /// A task that represents the asynchronous operation. + public Task InitializeAsync() => Task.CompletedTask; + + /// Releases the resources held by this fixture asynchronously. + /// A task that represents the asynchronous operation. + public async Task DisposeAsync() + { + foreach (var s in _systems) + { + try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ } + } + } +}