feat(cluster): leader-auto-down decision logging + cold-start/flapping/partition/switch-off scenarios (ScadaBridge#106)
Every downing decision of LeaderAutoDowningProvider is now logged under a [leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable), unreachable observed + its window (Warning), skipped because already Down/Exiting, pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own AutoDown logged only the final Down. Decision semantics are unchanged. New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests), built from production BuildHocon with self-first seeds: - both nodes cold-start together -> one cluster, nobody downed past FD + window; - lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up, then the peer cold-starts and joins; - flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the window, repeated past it in total -> never downed; positive control asserts the cuts really produced unreachability; - partition held past the window -> each side downs the other, both stay Up and leader (the accepted dual-active trade); - switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider, only leader-auto-down loads the in-repo provider. Unit: flapping never downs; a flapping member that finally stays unreachable is downed exactly once a full window after its LAST unreachable; each decision log line. Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping test fail; DOWNING log dropped -> log test fails; provider never downs -> partition test fails; auto-down miswired to the new provider -> switch-off theory fails. Docker failover drill not run (it would rebuild the shared scadabridge:latest image and reuse the shared rig volumes while another agent works in the repo); filed as deploy gate ScadaBridge#116. Default remains auto-down. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -199,7 +199,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
|
||||
### Cluster & Failover
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it.
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix.
|
||||
- Both nodes are seed nodes. `min-nr-of-members = 1`.
|
||||
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
||||
- CoordinatedShutdown for graceful singleton handover.
|
||||
|
||||
@@ -8,6 +8,16 @@
|
||||
`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value.
|
||||
Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the
|
||||
docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.
|
||||
|
||||
**Update 2026-11-10 (ScadaBridge#106 complete):** every downing decision is now logged with a `[leader-auto-down]`
|
||||
prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed,
|
||||
timed out on a non-leader → pending, leadership gained/lost, and `DOWNING` at Warning before `Cluster.Down`). New
|
||||
real-cluster scenarios in `IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs`: joint cold start, lone cold start
|
||||
with the peer dead then the peer joining, a flapping peer (Akka `trttl` throttler blackhole, unreachable shorter than the
|
||||
window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held
|
||||
past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (`auto-down` →
|
||||
`AutoDowning`, `keep-oldest` → `SplitBrainResolverProvider`). The docker drill and rollout are tracked as the deploy gate
|
||||
**ScadaBridge#116**.
|
||||
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
|
||||
|
||||
## 1. Current state (read from the code)
|
||||
|
||||
@@ -102,7 +102,7 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||
|
||||
|
||||
@@ -119,7 +119,6 @@ public sealed class LeaderAutoDowningProvider : IDowningProvider
|
||||
internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||
{
|
||||
private readonly Akka.Cluster.Cluster _cluster;
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
/// <summary>Props for the downing actor.</summary>
|
||||
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||
@@ -165,7 +164,6 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||
throw new InvalidOperationException("Must be leader to down node");
|
||||
}
|
||||
|
||||
_log.Info("Leader is auto-downing unreachable node [{0}]", node);
|
||||
_cluster.Down(node);
|
||||
}
|
||||
}
|
||||
@@ -174,6 +172,15 @@ internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||
/// The cluster-free decision logic of <see cref="LeaderAutoDown"/> — port of Akka's internal
|
||||
/// <c>AutoDownBase</c>, split out (as Akka does) so the leader / reachability semantics can be
|
||||
/// unit tested by feeding cluster events directly.
|
||||
/// <para>
|
||||
/// Every downing decision is logged (ScadaBridge#106) — with the <c>[leader-auto-down]</c>
|
||||
/// prefix so an operator can grep one node's log for the whole story: a peer observed
|
||||
/// unreachable (and the window it now has), a peer skipped because it is already Down/Exiting,
|
||||
/// a pending timer cancelled because the peer became reachable again or was removed, a timer
|
||||
/// that fired on a non-leader (kept pending), leadership gained/lost, and the actual Down
|
||||
/// (logged at Warning, before <see cref="Down"/> is called). Akka's own <c>AutoDown</c> logs
|
||||
/// only the last of these.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
{
|
||||
@@ -193,6 +200,9 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
ImmutableDictionary<UniqueAddress, ICancelable>.Empty;
|
||||
private ImmutableHashSet<UniqueAddress> _pendingUnreachable = ImmutableHashSet<UniqueAddress>.Empty;
|
||||
|
||||
/// <summary>Decision log; every downing decision is written here.</summary>
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
|
||||
/// <summary>Creates the base logic.</summary>
|
||||
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||
protected LeaderAutoDownBase(TimeSpan unreachableAfter)
|
||||
@@ -229,6 +239,10 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
{
|
||||
case ClusterEvent.CurrentClusterState state:
|
||||
IsLeader = state.Leader != null && state.Leader.Equals(SelfAddress);
|
||||
_log.Info(
|
||||
"[leader-auto-down] started on {0}: leader={1} (this node is leader: {2}), unreachable-after={3}, already unreachable=[{4}]",
|
||||
SelfAddress, state.Leader?.ToString() ?? "none", IsLeader, _unreachableAfter,
|
||||
string.Join(", ", state.Unreachable.Select(m => m.Address)));
|
||||
foreach (var member in state.Unreachable)
|
||||
{
|
||||
UnreachableMember(member);
|
||||
@@ -240,19 +254,31 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
break;
|
||||
|
||||
case ClusterEvent.ReachableMember reachable:
|
||||
Remove(reachable.Member.UniqueAddress);
|
||||
Remove(reachable.Member.UniqueAddress, "became reachable again");
|
||||
break;
|
||||
|
||||
case ClusterEvent.MemberRemoved removed:
|
||||
Remove(removed.Member.UniqueAddress);
|
||||
Remove(removed.Member.UniqueAddress, "was removed from the cluster");
|
||||
break;
|
||||
|
||||
case ClusterEvent.LeaderChanged leaderChanged:
|
||||
var wasLeader = IsLeader;
|
||||
IsLeader = leaderChanged.Leader != null && leaderChanged.Leader.Equals(SelfAddress);
|
||||
if (wasLeader != IsLeader)
|
||||
{
|
||||
_log.Info(
|
||||
"[leader-auto-down] {0} {1} leadership (leader is now {2}); pending unreachable=[{3}]",
|
||||
SelfAddress, IsLeader ? "gained" : "lost", leaderChanged.Leader?.ToString() ?? "none",
|
||||
string.Join(", ", _pendingUnreachable.Select(n => n.Address)));
|
||||
}
|
||||
|
||||
if (IsLeader)
|
||||
{
|
||||
foreach (var node in _pendingUnreachable)
|
||||
{
|
||||
_log.Warning(
|
||||
"[leader-auto-down] DOWNING {0}: it timed out as unreachable while this node was not leader, and this node ({1}) is now leader",
|
||||
node.Address, SelfAddress);
|
||||
Down(node.Address);
|
||||
}
|
||||
|
||||
@@ -274,7 +300,15 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
|
||||
private void UnreachableMember(Member member)
|
||||
{
|
||||
if (!SkipMemberStatus.Contains(member.Status) && !_scheduledUnreachable.ContainsKey(member.UniqueAddress))
|
||||
if (SkipMemberStatus.Contains(member.Status))
|
||||
{
|
||||
_log.Info(
|
||||
"[leader-auto-down] {0} is unreachable but already {1}; not scheduling a Down",
|
||||
member.Address, member.Status);
|
||||
return;
|
||||
}
|
||||
|
||||
if (!_scheduledUnreachable.ContainsKey(member.UniqueAddress))
|
||||
{
|
||||
ScheduleUnreachable(member.UniqueAddress);
|
||||
}
|
||||
@@ -288,6 +322,10 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
return;
|
||||
}
|
||||
|
||||
_log.Warning(
|
||||
"[leader-auto-down] {0} is unreachable; it will be downed if still unreachable after {1} (this node {2} is leader: {3})",
|
||||
node.Address, _unreachableAfter, SelfAddress, IsLeader);
|
||||
|
||||
var cancelable = Scheduler.ScheduleTellOnceCancelable(
|
||||
_unreachableAfter, Self, new UnreachableTimeout(node), Self);
|
||||
_scheduledUnreachable = _scheduledUnreachable.Add(node, cancelable);
|
||||
@@ -297,19 +335,31 @@ internal abstract class LeaderAutoDownBase : UntypedActor
|
||||
{
|
||||
if (IsLeader)
|
||||
{
|
||||
_log.Warning(
|
||||
"[leader-auto-down] DOWNING {0}: unreachable for {1} and this node ({2}) is leader",
|
||||
node.Address, _unreachableAfter, SelfAddress);
|
||||
Down(node.Address);
|
||||
}
|
||||
else
|
||||
{
|
||||
_log.Info(
|
||||
"[leader-auto-down] {0} unreachable for {1} but this node ({2}) is not leader; kept pending until it becomes leader",
|
||||
node.Address, _unreachableAfter, SelfAddress);
|
||||
_pendingUnreachable = _pendingUnreachable.Add(node);
|
||||
}
|
||||
}
|
||||
|
||||
private void Remove(UniqueAddress node)
|
||||
private void Remove(UniqueAddress node, string reason)
|
||||
{
|
||||
if (_scheduledUnreachable.TryGetValue(node, out var cancelable))
|
||||
var wasScheduled = _scheduledUnreachable.TryGetValue(node, out var cancelable);
|
||||
if (wasScheduled)
|
||||
{
|
||||
cancelable.Cancel();
|
||||
cancelable!.Cancel();
|
||||
}
|
||||
|
||||
if (wasScheduled || _pendingUnreachable.Contains(node))
|
||||
{
|
||||
_log.Info("[leader-auto-down] {0} {1}; its pending Down is cancelled", node.Address, reason);
|
||||
}
|
||||
|
||||
_scheduledUnreachable = _scheduledUnreachable.Remove(node);
|
||||
|
||||
@@ -18,6 +18,12 @@ namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||
/// </summary>
|
||||
public class LeaderAutoDownTests : TestKit
|
||||
{
|
||||
/// <summary>INFO so the decision-log assertions can see the Info-level decisions.</summary>
|
||||
public LeaderAutoDownTests()
|
||||
: base("akka.loglevel = INFO")
|
||||
{
|
||||
}
|
||||
|
||||
private static readonly Address MemberA = new("akka.tcp", "sys", "a", 2552);
|
||||
private static readonly Address MemberB = new("akka.tcp", "sys", "b", 2552);
|
||||
private static readonly Address MemberC = new("akka.tcp", "sys", "c", 2552);
|
||||
@@ -212,4 +218,107 @@ public class LeaderAutoDownTests : TestKit
|
||||
ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3));
|
||||
ExpectNoMsg(TimeSpan.FromSeconds(2));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A flapping member — unreachable, reachable again inside the window, repeatedly, for far
|
||||
/// longer than one window in total — is never downed.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void FlappingMember_ReachableAgainInsideEachWindow_IsNeverDowned()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.FromMilliseconds(1500));
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||
for (var i = 0; i < 5; i++)
|
||||
{
|
||||
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(700));
|
||||
a.Tell(new ClusterEvent.ReachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(100));
|
||||
}
|
||||
|
||||
ExpectNoMsg(TimeSpan.FromSeconds(2));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// When a flapping member finally stays unreachable, the window restarts from its LAST
|
||||
/// unreachable — an earlier flap's timer must not fire — and it is downed exactly once.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void FlappingMember_ThatStaysUnreachable_IsDownedOneFullWindowAfterLastUnreachable()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.FromMilliseconds(1500));
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
a.Tell(new ClusterEvent.ReachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
a.Tell(new ClusterEvent.ReachableMember(b));
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(200));
|
||||
a.Tell(new ClusterEvent.UnreachableMember(b)); // final: stays unreachable from here
|
||||
|
||||
// The first flap's timer would fire ~0.1s from here; the final window ends ~1.5s from here.
|
||||
ExpectNoMsg(TimeSpan.FromMilliseconds(1100));
|
||||
ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(2));
|
||||
ExpectNoMsg(TimeSpan.FromSeconds(2));
|
||||
}
|
||||
|
||||
/// <summary>The Down decision is logged at Warning, naming the member and the window.</summary>
|
||||
[Fact]
|
||||
public void DecisionLog_Down_IsLoggedAtWarning()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.Zero);
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
EventFilter.Warning(contains: $"[leader-auto-down] DOWNING {MemberB}").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up))));
|
||||
ExpectMsg(new DownCalled(MemberB));
|
||||
}
|
||||
|
||||
/// <summary>Observing an unreachable member (and its window) is logged at Warning.</summary>
|
||||
[Fact]
|
||||
public void DecisionLog_UnreachableScheduled_IsLoggedAtWarning()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.FromSeconds(30));
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
EventFilter.Warning(contains: $"{MemberB} is unreachable; it will be downed if still unreachable after").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up))));
|
||||
}
|
||||
|
||||
/// <summary>A cancelled Down (member reachable again) is logged.</summary>
|
||||
[Fact]
|
||||
public void DecisionLog_ReachableAgain_CancelIsLogged()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.FromSeconds(30));
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||
EventFilter.Info(contains: $"{MemberB} became reachable again; its pending Down is cancelled").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.ReachableMember(b)));
|
||||
}
|
||||
|
||||
/// <summary>A timeout on a non-leader (kept pending) is logged, and so is the later Down on becoming leader.</summary>
|
||||
[Fact]
|
||||
public void DecisionLog_PendingOnNonLeader_ThenDownOnBecomingLeader_AreLogged()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.Zero);
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||
EventFilter.Info(contains: "is not leader; kept pending").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up))));
|
||||
EventFilter.Warning(contains: $"DOWNING {MemberC}: it timed out as unreachable while this node was not leader").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA)));
|
||||
ExpectMsg(new DownCalled(MemberC));
|
||||
}
|
||||
|
||||
/// <summary>Skipping an already Down member is logged.</summary>
|
||||
[Fact]
|
||||
public void DecisionLog_SkippedDownMember_IsLogged()
|
||||
{
|
||||
var a = AutoDownActor(TimeSpan.Zero);
|
||||
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||
EventFilter.Info(contains: "but already Down; not scheduling a Down").ExpectOne(() =>
|
||||
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Down))));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,257 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Configuration;
|
||||
using Akka.Remote;
|
||||
using Akka.Remote.Transport;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
using ZB.MOM.WW.ScadaBridge.Host;
|
||||
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#106: the operational scenarios of the in-repo <see cref="LeaderAutoDowningProvider"/>
|
||||
/// beyond the hard-crash clones in <see cref="LeaderAutoDownFailoverTests"/> — cold starts
|
||||
/// (together, and alone with the peer dead: the shape that makes SBR <c>down-if-alone</c>
|
||||
/// self-down on a cold recreate), a FLAPPING peer that keeps going unreachable for less than
|
||||
/// the window, a real partition (the accepted dual-active trade), and the opt-in switch being
|
||||
/// off (the old providers are what Akka loads).
|
||||
/// <para>
|
||||
/// Nodes are built from the production <c>BuildHocon</c> output with the shipped self-first
|
||||
/// seed ordering, exactly like <see cref="SelfFirstSeedBootstrapTests"/>. The flapping and
|
||||
/// partition cases add Akka's own throttler transport adapter (<c>trttl</c>, the one the
|
||||
/// MultiNodeTestRunner uses for <c>Blackhole</c>) so the network can be cut and restored
|
||||
/// without killing either node.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class LeaderAutoDownScenarioTests : IAsyncLifetime
|
||||
{
|
||||
private const string Strategy = "leader-auto-down";
|
||||
private static readonly TimeSpan Heartbeat = TimeSpan.FromMilliseconds(500);
|
||||
private static readonly TimeSpan Threshold = TimeSpan.FromSeconds(2);
|
||||
|
||||
private readonly List<ActorSystem> _systems = new();
|
||||
|
||||
private ActorSystem StartNode(
|
||||
int selfPort, int peerPort, string strategy = Strategy, TimeSpan? stableAfter = null, bool throttled = false)
|
||||
{
|
||||
// With the throttler applied the published address scheme becomes akka.trttl.tcp, so the
|
||||
// seeds must carry it too or the nodes never find each other.
|
||||
var scheme = throttled ? "akka.trttl.tcp" : "akka.tcp";
|
||||
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
|
||||
var clusterOptions = new ClusterOptions
|
||||
{
|
||||
SeedNodes = new List<string>
|
||||
{
|
||||
$"{scheme}://scadabridge@127.0.0.1:{selfPort}",
|
||||
$"{scheme}://scadabridge@127.0.0.1:{peerPort}",
|
||||
},
|
||||
SplitBrainResolverStrategy = strategy,
|
||||
StableAfter = stableAfter ?? TimeSpan.FromSeconds(3),
|
||||
HeartbeatInterval = Heartbeat,
|
||||
FailureDetectionThreshold = Threshold,
|
||||
MinNrOfMembers = 1,
|
||||
DownIfAlone = true,
|
||||
};
|
||||
// Transport failure detection far above the cluster window: a blackhole must make the
|
||||
// MEMBER unreachable (what the downing provider reacts to), not tear the association down.
|
||||
var hocon = AkkaHostedService.BuildHocon(
|
||||
nodeOptions, clusterOptions, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(60));
|
||||
var overrides = "akka.coordinated-shutdown.run-by-actor-system-terminate = off\n"
|
||||
+ (throttled ? "akka.remote.dot-netty.tcp.applied-adapters = [trttl]\n" : string.Empty);
|
||||
var config = ConfigurationFactory.ParseString(overrides)
|
||||
.WithFallback(ConfigurationFactory.ParseString(hocon));
|
||||
var system = ActorSystem.Create("scadabridge", config);
|
||||
_systems.Add(system);
|
||||
return system;
|
||||
}
|
||||
|
||||
private static Task<bool> Throttle(ActorSystem from, ActorSystem to, ThrottleMode mode) =>
|
||||
((RemoteActorRefProvider)((ExtendedActorSystem)from).Provider).Transport.ManagementCommand(
|
||||
new SetThrottle(Akka.Cluster.Cluster.Get(to).SelfAddress, ThrottleTransportAdapter.Direction.Both, mode));
|
||||
|
||||
private static void AssertTwoUpNoneUnreachable(ActorSystem node, string because)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(node);
|
||||
Assert.False(cluster.IsTerminated, $"{cluster.SelfAddress}: Cluster extension terminated ({because})");
|
||||
Assert.Equal(MemberStatus.Up, cluster.SelfMember.Status);
|
||||
Assert.Equal(2, cluster.State.Members.Count(m => m.Status == MemberStatus.Up));
|
||||
Assert.Empty(cluster.State.Unreachable);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Both nodes cold-start together: they converge on ONE two-member cluster and, once
|
||||
/// formed, nobody is downed — held for well past failure detection plus the window.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task BothNodesColdStartTogether_FormOneCluster_AndNeitherIsDowned()
|
||||
{
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var nodeA = StartNode(portA, portB);
|
||||
var nodeB = StartNode(portB, portA);
|
||||
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60));
|
||||
Assert.IsType<LeaderAutoDowningProvider>(Akka.Cluster.Cluster.Get(nodeA).DowningProvider);
|
||||
|
||||
await Task.Delay(Threshold + TimeSpan.FromSeconds(3) + TimeSpan.FromSeconds(5));
|
||||
AssertTwoUpNoneUnreachable(nodeA, "after a joint cold start");
|
||||
AssertTwoUpNoneUnreachable(nodeB, "after a joint cold start");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A lone cold start with the peer DEAD (the cold-recreate shape where SBR keep-oldest
|
||||
/// <c>down-if-alone</c> downs the lone node) forms a one-node cluster that stays Up past the
|
||||
/// window; when the peer is later cold-started it joins, and nobody is downed.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task LoneColdStart_PeerDead_StaysUp_ThenPeerColdStartsAndJoins()
|
||||
{
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening yet
|
||||
var nodeA = StartNode(portA, portB);
|
||||
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
|
||||
await Task.Delay(Threshold + TimeSpan.FromSeconds(3) + TimeSpan.FromSeconds(3));
|
||||
var clusterA = Akka.Cluster.Cluster.Get(nodeA);
|
||||
Assert.False(clusterA.IsTerminated, "lone cold-started node downed itself");
|
||||
Assert.Equal(MemberStatus.Up, clusterA.SelfMember.Status);
|
||||
|
||||
var nodeB = StartNode(portB, portA);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60));
|
||||
await Task.Delay(TimeSpan.FromSeconds(5));
|
||||
AssertTwoUpNoneUnreachable(nodeA, "after the peer joined");
|
||||
AssertTwoUpNoneUnreachable(nodeB, "after the peer joined");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A FLAPPING peer — cut off long enough to be marked unreachable, restored before the
|
||||
/// window ends, over and over, for far longer than the window in total — is never downed:
|
||||
/// each return to reachable cancels its pending Down and the next cut starts a fresh window.
|
||||
/// The test first proves the peer really did go unreachable on (almost) every cut, so the
|
||||
/// "never downed" outcome is not vacuous.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task FlappingPeer_UnreachableShorterThanWindow_IsNeverDowned()
|
||||
{
|
||||
var stableAfter = TimeSpan.FromSeconds(8);
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var nodeA = StartNode(portA, portB, stableAfter: stableAfter, throttled: true);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
|
||||
var nodeB = StartNode(portB, portA, stableAfter: stableAfter, throttled: true);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(30));
|
||||
|
||||
var clusterA = Akka.Cluster.Cluster.Get(nodeA);
|
||||
const int flaps = 4;
|
||||
var flapsSeenUnreachable = 0;
|
||||
var started = DateTime.UtcNow;
|
||||
for (var i = 0; i < flaps; i++)
|
||||
{
|
||||
Assert.True(await Throttle(nodeA, nodeB, Blackhole.Instance), "SetThrottle(Blackhole) rejected");
|
||||
var seen = false;
|
||||
var cutUntil = DateTime.UtcNow + TimeSpan.FromSeconds(4.5);
|
||||
while (DateTime.UtcNow < cutUntil)
|
||||
{
|
||||
seen |= clusterA.State.Unreachable.Any();
|
||||
await Task.Delay(100);
|
||||
}
|
||||
|
||||
Assert.True(await Throttle(nodeA, nodeB, Unthrottled.Instance), "SetThrottle(Unthrottled) rejected");
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(15));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(15));
|
||||
if (seen)
|
||||
{
|
||||
flapsSeenUnreachable++;
|
||||
}
|
||||
}
|
||||
|
||||
// Positive control: the cuts really produced unreachability, and the flapping lasted
|
||||
// longer than one window — a provider that did not cancel on reachable would have downed.
|
||||
Assert.True(flapsSeenUnreachable >= flaps - 1,
|
||||
$"only {flapsSeenUnreachable}/{flaps} cuts made the peer unreachable — the flap never happened");
|
||||
Assert.True(DateTime.UtcNow - started > stableAfter * 2);
|
||||
|
||||
await Task.Delay(stableAfter + TimeSpan.FromSeconds(2));
|
||||
AssertTwoUpNoneUnreachable(nodeA, "flapping peer");
|
||||
AssertTwoUpNoneUnreachable(nodeB, "flapping peer");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A real partition longer than the window: each side is leader of its own reachable set,
|
||||
/// so each downs the other and BOTH stay Up — the dual-active outcome the availability-first
|
||||
/// decision accepts (identical to AutoDowning). Also the positive control for the flapping
|
||||
/// test: the same cut, held past the window, does down.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp()
|
||||
{
|
||||
var stableAfter = TimeSpan.FromSeconds(3);
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var nodeA = StartNode(portA, portB, stableAfter: stableAfter, throttled: true);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
|
||||
var nodeB = StartNode(portB, portA, stableAfter: stableAfter, throttled: true);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(30));
|
||||
var addressA = Akka.Cluster.Cluster.Get(nodeA).SelfAddress;
|
||||
var addressB = Akka.Cluster.Cluster.Get(nodeB).SelfAddress;
|
||||
|
||||
Assert.True(await Throttle(nodeA, nodeB, Blackhole.Instance));
|
||||
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(nodeA, addressB, TimeSpan.FromSeconds(30));
|
||||
await TwoNodeClusterFixture.WaitForMemberRemoved(nodeB, addressA, TimeSpan.FromSeconds(30));
|
||||
foreach (var node in new[] { nodeA, nodeB })
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(node);
|
||||
Assert.False(cluster.IsTerminated, $"{cluster.SelfAddress} downed itself during the partition");
|
||||
Assert.Equal(MemberStatus.Up, cluster.SelfMember.Status);
|
||||
Assert.Equal(cluster.SelfAddress, cluster.State.Leader);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The opt-in switch OFF: the default <c>auto-down</c> still loads Akka's own
|
||||
/// <c>AutoDowning</c> and <c>keep-oldest</c> still loads the SBR provider — the in-repo
|
||||
/// provider is loaded only for <c>leader-auto-down</c>.
|
||||
/// </summary>
|
||||
/// <param name="strategy">The configured strategy.</param>
|
||||
/// <param name="expectedProvider">The downing provider Akka must have instantiated.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Theory]
|
||||
[InlineData("auto-down", typeof(AutoDowning))]
|
||||
[InlineData("keep-oldest", typeof(Akka.Cluster.SBR.SplitBrainResolverProvider))]
|
||||
[InlineData(Strategy, typeof(LeaderAutoDowningProvider))]
|
||||
public async Task SwitchSelectsProvider_OldPathsUnchangedWhenOff(string strategy, Type expectedProvider)
|
||||
{
|
||||
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
|
||||
var nodeA = StartNode(portA, portB, strategy: strategy);
|
||||
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
|
||||
|
||||
Assert.IsType(expectedProvider, Akka.Cluster.Cluster.Get(nodeA).DowningProvider);
|
||||
}
|
||||
|
||||
/// <summary>Prepares the fixture before the tests run.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public Task InitializeAsync() => Task.CompletedTask;
|
||||
|
||||
/// <summary>Releases the resources held by this fixture asynchronously.</summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public async Task DisposeAsync()
|
||||
{
|
||||
foreach (var s in _systems)
|
||||
{
|
||||
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user