diff --git a/CLAUDE.md b/CLAUDE.md index d397feb2..838deb50 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -197,7 +197,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*. ### Cluster & Failover -- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). +- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it. - Both nodes are seed nodes. `min-nr-of-members = 1`. - Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`). - CoordinatedShutdown for graceful singleton handover. diff --git a/docs/plans/2026-11-09-autodowning-replacement-plan.md b/docs/plans/2026-11-09-autodowning-replacement-plan.md index f2a1996d..26caf234 100644 --- a/docs/plans/2026-11-09-autodowning-replacement-plan.md +++ b/docs/plans/2026-11-09-autodowning-replacement-plan.md @@ -1,6 +1,13 @@ # Replacing Akka `AutoDowning` before an Akka.NET upgrade removes it (plan, 2026-11-09) **Status: PLAN ONLY (ScadaBridge#51, register row 70).** No config or code changes are made by this document. + +**Update 2026-11-09 (ScadaBridge#106):** option B is implemented as the opt-in strategy `leader-auto-down` +(`LeaderAutoDowningProvider`, placed in **Host** rather than ClusterInfrastructure: the Code Placement rule in +`Component-ClusterInfrastructure.md` keeps ClusterInfrastructure Akka-free, and the provider is runtime wiring). +`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value. +Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the +docker failover drill (section 6) on the new value, then the per-environment rollout in section 5. Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md). ## 1. Current state (read from the code) diff --git a/docs/requirements/Component-ClusterInfrastructure.md b/docs/requirements/Component-ClusterInfrastructure.md index d47fad96..4c073e22 100644 --- a/docs/requirements/Component-ClusterInfrastructure.md +++ b/docs/requirements/Component-ClusterInfrastructure.md @@ -29,7 +29,7 @@ realised across two projects: timings, split-brain settings) bound from `appsettings.json` via the Options pattern. - **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`, - configures the downing strategy (`auto-down` by default; `keep-oldest` with `down-if-alone` is the opt-in alternative), wires + configures the downing strategy (`auto-down` by default; `leader-auto-down`, the in-repo port, and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires `CoordinatedShutdown` into the service lifecycle, and provides active-node / cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail. @@ -100,6 +100,10 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate - **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition. - **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips. +### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default) + +Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`. + ### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21) - **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability. diff --git a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs index b104bed1..5d3cefcf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs @@ -50,7 +50,7 @@ public class ClusterOptions public List SeedNodes { get; set; } = new(); /// - /// Downing strategy for unreachable members. Two supported values: + /// Downing strategy for unreachable members. Three supported values: /// /// auto-down (default, decision 2026-07-21) — availability-first: each /// side downs the unreachable peer after , so a hard crash @@ -59,6 +59,12 @@ public class ClusterOptions /// an operator restarts one side. Chosen because ScadaBridge pairs run one node per /// VM with no shared lease infrastructure, and a stalled system is a bigger risk /// than a rare partition. + /// leader-auto-down (opt-in, ScadaBridge#106) — identical semantics to + /// auto-down, implemented by the in-repo LeaderAutoDowningProvider (Host) instead + /// of Akka's deprecated AutoDowning, so it survives that provider's removal from a + /// future Akka.NET and does not log the boot-time deprecation warning. Not yet the default: + /// the docker failover drill must pass on it first + /// (docs/plans/2026-11-09-autodowning-replacement-plan.md). /// keep-oldest — partition-safe SBR: downs the side without the oldest /// member. In a TWO-node cluster this makes a crash of the oldest/active node a /// total outage: Akka's down-if-alone only rescues the survivor when its own diff --git a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs index acca52d3..28a39816 100644 --- a/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs @@ -20,10 +20,13 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBasestatic-quorum quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll /// on any unreachability in a 2-node cluster) and keep-majority keys the /// fatal crash to the lowest-address node instead of the oldest. + /// leader-auto-down (ScadaBridge#106) is the in-repo port of auto-down + /// (same semantics, no dependency on Akka's deprecated AutoDowning). /// private static readonly HashSet AllowedStrategies = new(StringComparer.OrdinalIgnoreCase) { "auto-down", + "leader-auto-down", "keep-oldest" }; @@ -48,7 +51,7 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase — the leader among the REACHABLE members /// downs the unreachable peer, so a crash of either node (oldest included) fails /// over to the survivor; the accepted trade is dual-active during a real network - /// partition. Any other value takes the SBR path, where the + /// partition. leader-auto-down (ScadaBridge#106, opt-in) installs the in-repo + /// — the same semantics without Akka's deprecated + /// provider — with its duration at + /// . + /// Any other value takes the SBR path, where the /// keep-oldest down-if-alone flag is emitted from /// rather than hard-coded, so the bound /// configuration value is actually consumed. @@ -280,10 +284,17 @@ public class AkkaHostedService : IHostedService // accepted trade — decision 2026-07-21). Anything else: the SBR provider with // the configured active-strategy (keep-oldest), which is partition-safe but // cannot survive a crash of the oldest node in a two-node cluster. - var downingBlock = string.Equals( - clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase) + // + // leader-auto-down (ScadaBridge#106, opt-in): the in-repo LeaderAutoDowningProvider — a + // port of AutoDowning with identical semantics that survives AutoDowning's eventual + // removal from Akka.NET. Its duration lives under our own key, so no + // auto-down-unreachable-after is emitted (which also silences Akka's deprecation warning). + var strategy = clusterOptions.SplitBrainResolverStrategy; + var downingBlock = string.Equals(strategy, "auto-down", StringComparison.OrdinalIgnoreCase) ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster"" auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}" + : string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase) + ? $@"downing-provider-class = {QuoteHocon(LeaderAutoDowningProvider.ProviderClassName)}" : $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"" split-brain-resolver {{ active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)} @@ -293,8 +304,15 @@ public class AkkaHostedService : IHostedService }} }}"; + // The leader-auto-down duration sits outside akka.cluster so Akka never mistakes it + // for one of its own (deprecated) auto-down keys. Only emitted for that strategy, so + // the auto-down / keep-oldest documents are byte-identical to before. + var leaderAutoDownBlock = string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase) + ? $"{LeaderAutoDowningProvider.UnreachableAfterPath} = {DurationHocon(clusterOptions.StableAfter)}\n" + : string.Empty; + return $@" -audit-telemetry-dispatcher {{ +{leaderAutoDownBlock}audit-telemetry-dispatcher {{ type = ForkJoinDispatcher throughput = 100 dedicated-thread-pool {{ diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs new file mode 100644 index 00000000..37f4229b --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs @@ -0,0 +1,318 @@ +using System.Collections.Immutable; +using Akka.Actor; +using Akka.Cluster; +using Akka.Configuration; +using Akka.Event; + +namespace ZB.MOM.WW.ScadaBridge.Host.Actors; + +/// +/// In-repo replacement for Akka.NET's deprecated Akka.Cluster.AutoDowning provider +/// (ScadaBridge#51 plan, implemented by ScadaBridge#106). Selected by +/// ScadaBridge:Cluster:SplitBrainResolverStrategy = "leader-auto-down"; the HOCON is +/// emitted by . +/// +/// Behaviour is a line-for-line port of Akka.NET 1.5.62 AutoDowning / +/// AutoDown / AutoDownBase (read from the decompiled 1.5.62 assembly), so +/// no operational contract changes when a pair switches from auto-down: the leader +/// among the REACHABLE members downs a peer that has stayed unreachable for +/// unreachable-after (= ClusterOptions.StableAfter); a non-leader records the +/// peer as pending and downs it if it later becomes leader; a peer that becomes reachable +/// again, or is removed, before the timer fires is forgotten. Because Akka computes the +/// leader over reachable members only, a crash of EITHER node of a pair (oldest included) +/// makes the survivor leader and it downs the crashed peer — the availability-first posture +/// of the 2026-07-21 decision, with dual-active on a real partition as the accepted trade. +/// +/// +/// Differences from AutoDowning, all deliberate and none behavioural: +/// the duration is read from rather than the deprecated +/// akka.cluster.auto-down-unreachable-after key (so Akka's boot-time deprecation +/// warning — "keep-majority will be used instead" — is no longer logged), and the +/// is read from akka.cluster.down-removal-margin +/// directly because ClusterSettings.DownRemovalMargin is internal (same off-switch +/// parsing; default off = zero, exactly what AutoDowning returns). +/// +/// +/// Akka instantiates this by reflection (Akka.Cluster.DowningProvider.Load) with the +/// (ActorSystem, Cluster) constructor; the downing actor is created as +/// /system/cluster/core/daemon/downingProvider. +/// +/// +public sealed class LeaderAutoDowningProvider : IDowningProvider +{ + /// HOCON path carrying the unreachable-before-down duration. + public const string UnreachableAfterPath = "scadabridge.cluster.leader-auto-down.unreachable-after"; + + /// Assembly-qualified type name for akka.cluster.downing-provider-class. + public static readonly string ProviderClassName = + $"{typeof(LeaderAutoDowningProvider).FullName}, {typeof(LeaderAutoDowningProvider).Assembly.GetName().Name}"; + + private readonly ActorSystem _system; + private readonly Akka.Cluster.Cluster _cluster; + + /// Creates the provider; called by Akka via reflection. + /// The actor system the cluster runs in. + /// The cluster extension being started. + public LeaderAutoDowningProvider(ActorSystem system, Akka.Cluster.Cluster cluster) + { + _system = system; + _cluster = cluster; + } + + /// + public TimeSpan DownRemovalMargin + { + get + { + var clusterConfig = _system.Settings.Config.GetConfig("akka.cluster"); + var raw = clusterConfig.GetString("down-removal-margin", string.Empty).ToLowerInvariant(); + return raw is "off" or "false" or "no" + ? TimeSpan.Zero + : clusterConfig.GetTimeSpan("down-removal-margin", null, true); + } + } + + /// + /// + /// is missing or off — mirrors AutoDowning's + /// refusal to start without auto-down-unreachable-after. + /// + public Props DowningActorProps + { + get + { + var after = ReadUnreachableAfter(_system.Settings.Config); + if (!after.HasValue) + { + throw new ConfigurationException( + $"LeaderAutoDowningProvider selected but '{UnreachableAfterPath}' not set"); + } + + return LeaderAutoDown.Props(after.Value, _cluster); + } + } + + /// + /// Reads with Akka's off-switch semantics (Akka's own + /// GetTimeSpanWithOffSwitch is internal): missing or off/false/no + /// yields . + /// + private static TimeSpan? ReadUnreachableAfter(Config config) + { + if (!config.HasPath(UnreachableAfterPath)) + { + return null; + } + + var raw = config.GetString(UnreachableAfterPath, string.Empty).ToLowerInvariant(); + return raw is "off" or "false" or "no" + ? null + : config.GetTimeSpan(UnreachableAfterPath, null, true); + } +} + +/// +/// The downing actor for — port of Akka's internal +/// AutoDown: subscribes to cluster domain events and downs through +/// . +/// +internal sealed class LeaderAutoDown : LeaderAutoDownBase +{ + private readonly Akka.Cluster.Cluster _cluster; + private readonly ILoggingAdapter _log = Context.GetLogger(); + + /// Props for the downing actor. + /// How long a member must stay unreachable before it is downed. + /// The cluster extension. + public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) => + Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster)); + + /// Creates the actor. + /// How long a member must stay unreachable before it is downed. + /// The cluster extension. + public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) + : base(unreachableAfter) + { + _cluster = cluster; + } + + /// + public override Address SelfAddress => _cluster.SelfAddress; + + /// + public override IScheduler Scheduler => Context.System.Scheduler; + + /// + protected override void PreStart() + { + _cluster.Subscribe(Self, typeof(ClusterEvent.IClusterDomainEvent)); + base.PreStart(); + } + + /// + protected override void PostStop() + { + _cluster.Unsubscribe(Self); + base.PostStop(); + } + + /// + /// Called on a non-leader (a logic error). + public override void Down(Address node) + { + if (!IsLeader) + { + throw new InvalidOperationException("Must be leader to down node"); + } + + _log.Info("Leader is auto-downing unreachable node [{0}]", node); + _cluster.Down(node); + } +} + +/// +/// The cluster-free decision logic of — port of Akka's internal +/// AutoDownBase, split out (as Akka does) so the leader / reachability semantics can be +/// unit tested by feeding cluster events directly. +/// +internal abstract class LeaderAutoDownBase : UntypedActor +{ + /// Timer message: has been unreachable for the full window. + /// The member that was scheduled for downing. + internal sealed record UnreachableTimeout(UniqueAddress Node); + + /// + /// Members in these states are not scheduled — Akka's + /// MembershipState.ConvergenceSkipUnreachableWithMemberStatus (Down, Exiting). + /// + private static readonly ImmutableHashSet SkipMemberStatus = + ImmutableHashSet.Create(MemberStatus.Down, MemberStatus.Exiting); + + private readonly TimeSpan _unreachableAfter; + private ImmutableDictionary _scheduledUnreachable = + ImmutableDictionary.Empty; + private ImmutableHashSet _pendingUnreachable = ImmutableHashSet.Empty; + + /// Creates the base logic. + /// How long a member must stay unreachable before it is downed. + protected LeaderAutoDownBase(TimeSpan unreachableAfter) + { + _unreachableAfter = unreachableAfter; + } + + /// Whether this node is currently the cluster leader. + protected bool IsLeader { get; private set; } + + /// This node's address, compared against the leader address. + public abstract Address SelfAddress { get; } + + /// The scheduler used for the unreachable timers. + public abstract IScheduler Scheduler { get; } + + /// Downs ; only ever called while leader. + /// The member to down. + public abstract void Down(Address node); + + /// + protected override void PostStop() + { + foreach (var cancelable in _scheduledUnreachable.Values) + { + cancelable.Cancel(); + } + } + + /// + protected override void OnReceive(object message) + { + switch (message) + { + case ClusterEvent.CurrentClusterState state: + IsLeader = state.Leader != null && state.Leader.Equals(SelfAddress); + foreach (var member in state.Unreachable) + { + UnreachableMember(member); + } + break; + + case ClusterEvent.UnreachableMember unreachable: + UnreachableMember(unreachable.Member); + break; + + case ClusterEvent.ReachableMember reachable: + Remove(reachable.Member.UniqueAddress); + break; + + case ClusterEvent.MemberRemoved removed: + Remove(removed.Member.UniqueAddress); + break; + + case ClusterEvent.LeaderChanged leaderChanged: + IsLeader = leaderChanged.Leader != null && leaderChanged.Leader.Equals(SelfAddress); + if (IsLeader) + { + foreach (var node in _pendingUnreachable) + { + Down(node.Address); + } + + _pendingUnreachable = ImmutableHashSet.Empty; + } + break; + + case UnreachableTimeout timeout: + if (_scheduledUnreachable.ContainsKey(timeout.Node)) + { + _scheduledUnreachable = _scheduledUnreachable.Remove(timeout.Node); + DownOrAddPending(timeout.Node); + } + break; + + // Every other IClusterDomainEvent is ignored, exactly as AutoDownBase does. + } + } + + private void UnreachableMember(Member member) + { + if (!SkipMemberStatus.Contains(member.Status) && !_scheduledUnreachable.ContainsKey(member.UniqueAddress)) + { + ScheduleUnreachable(member.UniqueAddress); + } + } + + private void ScheduleUnreachable(UniqueAddress node) + { + if (_unreachableAfter == TimeSpan.Zero) + { + DownOrAddPending(node); + return; + } + + var cancelable = Scheduler.ScheduleTellOnceCancelable( + _unreachableAfter, Self, new UnreachableTimeout(node), Self); + _scheduledUnreachable = _scheduledUnreachable.Add(node, cancelable); + } + + private void DownOrAddPending(UniqueAddress node) + { + if (IsLeader) + { + Down(node.Address); + } + else + { + _pendingUnreachable = _pendingUnreachable.Add(node); + } + } + + private void Remove(UniqueAddress node) + { + if (_scheduledUnreachable.TryGetValue(node, out var cancelable)) + { + cancelable.Cancel(); + } + + _scheduledUnreachable = _scheduledUnreachable.Remove(node); + _pendingUnreachable = _pendingUnreachable.Remove(node); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs index 88fecb41..41abedfb 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs @@ -298,4 +298,33 @@ public class ClusterOptionsValidatorTests Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage); Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage); } + + /// Verifies that the leader-auto-down strategy (ScadaBridge#106) passes validation, case-insensitively. + /// The strategy spelling. + [Theory] + [InlineData("leader-auto-down")] + [InlineData("Leader-Auto-Down")] + public void LeaderAutoDownStrategy_Passes(string strategy) + { + var options = ValidOptions(); + options.SplitBrainResolverStrategy = strategy; + options.DownIfAlone = false; // keep-oldest-only knob; inert here, must not block startup + + var result = new ClusterOptionsValidator().Validate(null, options); + + Assert.True(result.Succeeded, result.FailureMessage); + } + + /// Verifies that the rejection message lists leader-auto-down among the supported strategies. + [Fact] + public void UnsupportedStrategy_MessageNamesLeaderAutoDown() + { + var options = ValidOptions(); + options.SplitBrainResolverStrategy = "leader-autodown"; + + var result = new ClusterOptionsValidator().Validate(null, options); + + Assert.True(result.Failed); + Assert.Contains("'leader-auto-down'", result.FailureMessage); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs index c7852764..a1f93433 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/HoconBuilderTests.cs @@ -250,4 +250,76 @@ public class HoconBuilderTests Assert.Equal(1, config.GetInt("akka.cluster.min-nr-of-members")); Assert.True(config.GetBoolean("akka.cluster.run-coordinated-shutdown-when-down")); } + + /// + /// Verifies that the leader-auto-down strategy (ScadaBridge#106) installs the in-repo + /// provider, carries StableAfter under its own key, and emits neither Akka's deprecated + /// auto-down key nor an SBR section. + /// + [Fact] + public void BuildHocon_LeaderAutoDownStrategy_EmitsInRepoProvider_AndNoDeprecatedKey() + { + var cluster = DefaultCluster(); + cluster.SplitBrainResolverStrategy = "leader-auto-down"; + cluster.StableAfter = TimeSpan.FromMilliseconds(15250); + + var hocon = AkkaHostedService.BuildHocon( + DefaultNode(), cluster, new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)); + + var config = ConfigurationFactory.ParseString(hocon); + Assert.Equal( + "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host", + config.GetString("akka.cluster.downing-provider-class")); + // The emitted name must resolve the way Akka resolves it (Type.GetType(name, true)). + Assert.Equal(typeof(LeaderAutoDowningProvider), + Type.GetType(config.GetString("akka.cluster.downing-provider-class"), throwOnError: true)); + Assert.Equal( + TimeSpan.FromMilliseconds(15250), + config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath)); + // No deprecated key (that is what triggers Akka's "keep-majority will be used" warning) + // and no SBR section alongside the provider. + Assert.False(config.HasPath("akka.cluster.auto-down-unreachable-after")); + Assert.False(config.HasPath("akka.cluster.split-brain-resolver.active-strategy")); + // Document stays intact after the downing block. + Assert.Equal(1, config.GetInt("akka.cluster.min-nr-of-members")); + Assert.True(config.GetBoolean("akka.cluster.run-coordinated-shutdown-when-down")); + } + + /// Verifies that the leader-auto-down strategy name is case-insensitive. + [Fact] + public void BuildHocon_LeaderAutoDownStrategy_IsCaseInsensitive() + { + var cluster = DefaultCluster(); + cluster.SplitBrainResolverStrategy = "Leader-Auto-Down"; + + var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon( + DefaultNode(), cluster, new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15))); + + Assert.Equal(LeaderAutoDowningProvider.ProviderClassName, + config.GetString("akka.cluster.downing-provider-class")); + Assert.Equal(TimeSpan.FromSeconds(15), config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath)); + } + + /// + /// Verifies the leader-auto-down key is emitted ONLY for that strategy: the existing + /// auto-down and keep-oldest documents are unchanged by ScadaBridge#106. + /// + /// An existing strategy. + [Theory] + [InlineData("auto-down")] + [InlineData("keep-oldest")] + public void BuildHocon_OtherStrategies_DoNotEmitLeaderAutoDownKey(string strategy) + { + var cluster = DefaultCluster(); + cluster.SplitBrainResolverStrategy = strategy; + + var hocon = AkkaHostedService.BuildHocon( + DefaultNode(), cluster, new[] { "Central" }, + TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)); + + Assert.DoesNotContain("leader-auto-down", hocon); + Assert.DoesNotContain(nameof(LeaderAutoDowningProvider), hocon); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs new file mode 100644 index 00000000..ff408735 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/LeaderAutoDownTests.cs @@ -0,0 +1,215 @@ +using System.Collections.Immutable; +using System.Reflection; +using Akka.Actor; +using Akka.Cluster; +using Akka.TestKit.Xunit2; +using Akka.Util; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.Host.Tests; + +/// +/// ScadaBridge#106: the decision logic of must match Akka.NET +/// 1.5.62 AutoDownBase exactly. These cases are the ones Akka's own AutoDownSpec +/// pins — cluster events are fed straight to the actor and the Down seam reports to a +/// probe, so the leader-only and reachability semantics are proven without a running cluster. +/// The two-node real-cluster behaviour is proven separately in IntegrationTests +/// (LeaderAutoDownFailoverTests). +/// +public class LeaderAutoDownTests : TestKit +{ + private static readonly Address MemberA = new("akka.tcp", "sys", "a", 2552); + private static readonly Address MemberB = new("akka.tcp", "sys", "b", 2552); + private static readonly Address MemberC = new("akka.tcp", "sys", "c", 2552); + + private sealed record DownCalled(Address Address); + + /// Test double: Down reports to a probe instead of calling the cluster. + private sealed class TestLeaderAutoDown : LeaderAutoDownBase + { + private readonly IActorRef _probe; + + public TestLeaderAutoDown(Address selfAddress, TimeSpan unreachableAfter, IActorRef probe) + : base(unreachableAfter) + { + SelfAddress = selfAddress; + _probe = probe; + } + + public override Address SelfAddress { get; } + + public override IScheduler Scheduler => Context.System.Scheduler; + + public override void Down(Address node) + { + // Report EVERY call, leader or not: a base that wrongly downs while not leader must + // show up as an unexpected message, not be swallowed by a throw-and-restart. + _probe.Tell(new DownCalled(node)); + if (!IsLeader) + { + _probe.Tell("down-called-while-not-leader"); + } + } + } + + /// Akka's Member constructors are internal; tests build members via the internal factory. + private static Member CreateMember(Address address, MemberStatus status) + { + var create = typeof(Member).GetMethod( + "Create", + BindingFlags.Static | BindingFlags.NonPublic, + new[] { typeof(UniqueAddress), typeof(int), typeof(MemberStatus), typeof(ImmutableHashSet), typeof(AppVersion) }) + ?? throw new InvalidOperationException("Akka.Cluster.Member.Create(…) not found — Akka internals changed."); + return (Member)create.Invoke(null, new object[] + { + new UniqueAddress(address, 1), 1, status, ImmutableHashSet.Empty, AppVersion.Zero, + })!; + } + + private IActorRef AutoDownActor(TimeSpan unreachableAfter) => + Sys.ActorOf(Props.Create(() => new TestLeaderAutoDown(MemberA, unreachableAfter, TestActor))); + + /// A non-leader never downs an unreachable member. + [Fact] + public void NotLeader_DoesNotDownUnreachable() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up))); + ExpectNoMsg(TimeSpan.FromSeconds(1)); + } + + /// The leader downs an unreachable member immediately when the window is zero. + [Fact] + public void Leader_DownsUnreachable_WhenWindowIsZero() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up))); + ExpectMsg(new DownCalled(MemberB)); + } + + /// The leader downs an unreachable member only after the full window. + [Fact] + public void Leader_DownsUnreachable_AfterWindow() + { + var a = AutoDownActor(TimeSpan.FromSeconds(2)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up))); + ExpectNoMsg(TimeSpan.FromSeconds(1)); + ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3)); + } + + /// The initial snapshot drives leadership and schedules members already unreachable. + [Fact] + public void CurrentClusterState_Leader_DownsAlreadyUnreachable() + { + var a = AutoDownActor(TimeSpan.Zero); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.CurrentClusterState( + ImmutableSortedSet.Empty, + ImmutableHashSet.Create(b), + ImmutableHashSet
.Empty, + MemberA, + ImmutableDictionary.Empty)); + ExpectMsg(new DownCalled(MemberB)); + } + + /// A pending member is downed as soon as this node becomes leader. + [Fact] + public void BecomingLeader_DownsPendingUnreachable() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up))); + ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + ExpectMsg(new DownCalled(MemberC)); + } + + /// A member that timed out while not leader is downed on becoming leader. + [Fact] + public void BecomingLeader_AfterWindow_DownsPendingUnreachable() + { + var a = AutoDownActor(TimeSpan.FromSeconds(1)); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up))); + ExpectNoMsg(TimeSpan.FromSeconds(2)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + ExpectMsg(new DownCalled(MemberC)); + } + + /// Losing leadership inside the window means no Down when the timer fires. + [Fact] + public void LosingLeadership_InsideWindow_DoesNotDown() + { + var a = AutoDownActor(TimeSpan.FromSeconds(2)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up))); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + ExpectNoMsg(TimeSpan.FromSeconds(3)); + } + + /// A member that becomes reachable again inside the window is not downed. + [Fact] + public void ReachableAgain_InsideWindow_DoesNotDown() + { + var a = AutoDownActor(TimeSpan.FromSeconds(2)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(b)); + a.Tell(new ClusterEvent.ReachableMember(b)); + ExpectNoMsg(TimeSpan.FromSeconds(3)); + } + + /// A pending member that becomes reachable again is not downed on becoming leader. + [Fact] + public void ReachableAgain_WhilePending_IsForgotten() + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberB)); + var c = CreateMember(MemberC, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(c)); + a.Tell(new ClusterEvent.ReachableMember(c)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + ExpectNoMsg(TimeSpan.FromSeconds(1)); + } + + /// A member removed inside the window is not downed. + [Fact] + public void RemovedInsideWindow_DoesNotDown() + { + var a = AutoDownActor(TimeSpan.FromSeconds(2)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(b)); + a.Tell(new ClusterEvent.MemberRemoved(b.Copy(MemberStatus.Removed), MemberStatus.Up)); + ExpectNoMsg(TimeSpan.FromSeconds(3)); + } + + /// Members already Down or Exiting are skipped (Akka's ConvergenceSkipUnreachableWithMemberStatus). + /// The member status. + [Theory] + [InlineData(MemberStatus.Down)] + [InlineData(MemberStatus.Exiting)] + public void AlreadyDownOrExiting_IsNotDownedAgain(MemberStatus status) + { + var a = AutoDownActor(TimeSpan.Zero); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, status))); + ExpectNoMsg(TimeSpan.FromSeconds(1)); + } + + /// A repeated UnreachableMember for an already-scheduled member does not schedule a second Down. + [Fact] + public void RepeatedUnreachable_DownsOnce() + { + var a = AutoDownActor(TimeSpan.FromSeconds(1)); + a.Tell(new ClusterEvent.LeaderChanged(MemberA)); + var b = CreateMember(MemberB, MemberStatus.Up); + a.Tell(new ClusterEvent.UnreachableMember(b)); + a.Tell(new ClusterEvent.UnreachableMember(b)); + ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3)); + ExpectNoMsg(TimeSpan.FromSeconds(2)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownFailoverTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownFailoverTests.cs new file mode 100644 index 00000000..3186b4af --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/LeaderAutoDownFailoverTests.cs @@ -0,0 +1,123 @@ +using Akka.Actor; +using Xunit; +using ZB.MOM.WW.ScadaBridge.Host.Actors; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster; + +/// +/// ScadaBridge#106: the in-repo (strategy +/// leader-auto-down) must give a REAL two-node cluster exactly the outcomes Akka's +/// AutoDowning (auto-down) gives in : a hard crash +/// of EITHER node — the oldest included — is downed and removed by the survivor and the +/// singleton ends up answering on the survivor. The two failover cases are clones of +/// SbrFailoverTests.AutoDown_HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton and +/// …AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton, differing only in the strategy. +/// Under a provider that never downs, the member-removal step times out (the crashed member +/// stays Unreachable forever) — that is what gives them teeth. +/// +public class LeaderAutoDownFailoverTests +{ + private const string Strategy = "leader-auto-down"; + + /// + /// The configured provider is the one Akka actually instantiated and started: the Cluster + /// extension holds a and its downing actor is running + /// under the cluster daemon on both nodes. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task LeaderAutoDown_ProviderIsWhatAkkaLoads_OnBothNodes() + { + await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy); + + foreach (var node in new[] { cluster.NodeA, cluster.NodeB }) + { + var akkaCluster = Akka.Cluster.Cluster.Get(node); + Assert.IsType(akkaCluster.DowningProvider); + + // ClusterCoreDaemon creates the provider's DowningActorProps as "downingProvider". + var identity = await node.ActorSelection("/system/cluster/core/daemon/downingProvider") + .Ask(new Identify("downing"), TimeSpan.FromSeconds(5)); + Assert.NotNull(identity.Subject); + } + } + + /// + /// With leader-auto-down, a hard crash of the OLDEST node lets the younger survivor down it + /// and take over the singleton — the direction two-node keep-oldest cannot survive. + /// + /// A task that represents the asynchronous operation. + [Fact] + public async Task LeaderAutoDown_HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton() + { + await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy); + SbrFailoverTests.StartSingleton(cluster.NodeA); // oldest hosts the singleton initially + var (_, proxyB) = SbrFailoverTests.StartSingleton(cluster.NodeB); + + var echo = await proxyB.Ask("ping", TimeSpan.FromSeconds(20)); + Assert.Equal("ping", echo); + + var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress; + await TwoNodeClusterFixture.CrashNode(cluster.NodeA); + + // 1) The younger survivor must DOWN and REMOVE the crashed OLDEST member. + await TwoNodeClusterFixture.WaitForMemberRemoved( + cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30)); + + // 2) B must still be a functioning cluster member (not self-downed) … + var clusterB = Akka.Cluster.Cluster.Get(cluster.NodeB); + Assert.False(clusterB.IsTerminated, "survivor's Cluster extension terminated — it downed itself"); + + // 3) … and the singleton must migrate to B and answer again. + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30); + Exception? last = null; + while (DateTime.UtcNow < deadline) + { + try + { + var echo2 = await proxyB.Ask("ping-after-oldest-crash", TimeSpan.FromSeconds(3)); + Assert.Equal("ping-after-oldest-crash", echo2); + return; + } + catch (Exception ex) { last = ex; } + } + throw new Xunit.Sdk.XunitException( + $"Singleton never migrated to the younger survivor after the oldest crashed under leader-auto-down: {last}"); + } + + /// With leader-auto-down, a hard crash of the younger node leaves the singleton on the oldest node. + /// A task that represents the asynchronous operation. + [Fact] + public async Task LeaderAutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton() + { + await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy); + var (_, proxyA) = SbrFailoverTests.StartSingleton(cluster.NodeA); + SbrFailoverTests.StartSingleton(cluster.NodeB); + + Assert.Equal("ping", await proxyA.Ask("ping", TimeSpan.FromSeconds(20))); + + var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress; + await TwoNodeClusterFixture.CrashNode(cluster.NodeB); + + await TwoNodeClusterFixture.WaitForMemberRemoved( + cluster.NodeA, victimAddress, TimeSpan.FromSeconds(30)); + + Assert.False(Akka.Cluster.Cluster.Get(cluster.NodeA).IsTerminated, + "surviving oldest's Cluster extension terminated — it downed itself"); + + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30); + Exception? last = null; + while (DateTime.UtcNow < deadline) + { + try + { + var echo2 = await proxyA.Ask("ping-after-crash", TimeSpan.FromSeconds(3)); + Assert.Equal("ping-after-crash", echo2); + return; + } + catch (Exception ex) { last = ex; } + } + throw new Xunit.Sdk.XunitException( + $"Singleton stopped answering on the surviving oldest node under leader-auto-down: {last}"); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs index 33ec94bb..5e47d8c3 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SbrFailoverTests.cs @@ -34,7 +34,7 @@ public class SbrFailoverTests public EchoActor() => ReceiveAny(msg => Sender.Tell(msg)); } - private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys) + internal static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys) { var manager = sys.ActorOf(ClusterSingletonManager.Props( Props.Create(() => new EchoActor()),