feat(cluster): opt-in leader-auto-down strategy - in-repo port of Akka AutoDowning (ScadaBridge#106)
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET 1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the leader among the reachable members downs a peer unreachable for StableAfter; a non-leader records it as pending and downs it on becoming leader; reachable-again or removed members are forgotten; Down/Exiting members are skipped. Its window is carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no deprecated auto-down-unreachable-after key is emitted and Akka's boot-time "keep-majority will be used instead" warning goes away. DownRemovalMargin reads akka.cluster.down-removal-margin exactly as AutoDowning does (default off). New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon and ClusterOptionsValidator. The default stays "auto-down"; auto-down and keep-oldest HOCON is byte-identical to before. Placed in Host, not ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays Akka-free). Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec cases (leader-only, pending-on-leader-change, losing leadership, reachable again, removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node cluster plus a test that Akka actually loaded the provider and its downing actor. Mutation-checked: a provider that never calls Cluster.Down fails both failover tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed mutations each fail the unit tests. Remaining gate before switching the default: the docker failover drill (DRILL_MODE=active and standby mode) on leader-auto-down. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -197,7 +197,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
|||||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); prod (`vd03`, single node) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||||
|
|
||||||
### Cluster & Failover
|
### Cluster & Failover
|
||||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70).
|
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it.
|
||||||
- Both nodes are seed nodes. `min-nr-of-members = 1`.
|
- Both nodes are seed nodes. `min-nr-of-members = 1`.
|
||||||
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
||||||
- CoordinatedShutdown for graceful singleton handover.
|
- CoordinatedShutdown for graceful singleton handover.
|
||||||
|
|||||||
@@ -1,6 +1,13 @@
|
|||||||
# Replacing Akka `AutoDowning` before an Akka.NET upgrade removes it (plan, 2026-11-09)
|
# Replacing Akka `AutoDowning` before an Akka.NET upgrade removes it (plan, 2026-11-09)
|
||||||
|
|
||||||
**Status: PLAN ONLY (ScadaBridge#51, register row 70).** No config or code changes are made by this document.
|
**Status: PLAN ONLY (ScadaBridge#51, register row 70).** No config or code changes are made by this document.
|
||||||
|
|
||||||
|
**Update 2026-11-09 (ScadaBridge#106):** option B is implemented as the opt-in strategy `leader-auto-down`
|
||||||
|
(`LeaderAutoDowningProvider`, placed in **Host** rather than ClusterInfrastructure: the Code Placement rule in
|
||||||
|
`Component-ClusterInfrastructure.md` keeps ClusterInfrastructure Akka-free, and the provider is runtime wiring).
|
||||||
|
`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value.
|
||||||
|
Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the
|
||||||
|
docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.
|
||||||
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
|
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
|
||||||
|
|
||||||
## 1. Current state (read from the code)
|
## 1. Current state (read from the code)
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ realised across two projects:
|
|||||||
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
|
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
|
||||||
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
|
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
|
||||||
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
|
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
|
||||||
configures the downing strategy (`auto-down` by default; `keep-oldest` with `down-if-alone` is the opt-in alternative), wires
|
configures the downing strategy (`auto-down` by default; `leader-auto-down`, the in-repo port, and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
|
||||||
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
|
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
|
||||||
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
|
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
|
||||||
|
|
||||||
@@ -100,6 +100,10 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
|||||||
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||||
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
|
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
|
||||||
|
|
||||||
|
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||||
|
|
||||||
|
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||||
|
|
||||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||||
|
|
||||||
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
|
- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
|
||||||
|
|||||||
@@ -50,7 +50,7 @@ public class ClusterOptions
|
|||||||
public List<string> SeedNodes { get; set; } = new();
|
public List<string> SeedNodes { get; set; } = new();
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Downing strategy for unreachable members. Two supported values:
|
/// Downing strategy for unreachable members. Three supported values:
|
||||||
/// <list type="bullet">
|
/// <list type="bullet">
|
||||||
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
|
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
|
||||||
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
|
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
|
||||||
@@ -59,6 +59,12 @@ public class ClusterOptions
|
|||||||
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
|
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
|
||||||
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
|
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
|
||||||
/// than a rare partition.</item>
|
/// than a rare partition.</item>
|
||||||
|
/// <item><c>leader-auto-down</c> (opt-in, ScadaBridge#106) — identical semantics to
|
||||||
|
/// <c>auto-down</c>, implemented by the in-repo <c>LeaderAutoDowningProvider</c> (Host) instead
|
||||||
|
/// of Akka's deprecated <c>AutoDowning</c>, so it survives that provider's removal from a
|
||||||
|
/// future Akka.NET and does not log the boot-time deprecation warning. Not yet the default:
|
||||||
|
/// the docker failover drill must pass on it first
|
||||||
|
/// (<c>docs/plans/2026-11-09-autodowning-replacement-plan.md</c>).</item>
|
||||||
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
|
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
|
||||||
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
|
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
|
||||||
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
|
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
|
||||||
|
|||||||
@@ -20,10 +20,13 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
|||||||
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
|
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
|
||||||
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
|
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
|
||||||
/// fatal crash to the lowest-address node instead of the oldest.
|
/// fatal crash to the lowest-address node instead of the oldest.
|
||||||
|
/// <c>leader-auto-down</c> (ScadaBridge#106) is the in-repo port of <c>auto-down</c>
|
||||||
|
/// (same semantics, no dependency on Akka's deprecated <c>AutoDowning</c>).
|
||||||
/// </summary>
|
/// </summary>
|
||||||
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
|
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
|
||||||
{
|
{
|
||||||
"auto-down",
|
"auto-down",
|
||||||
|
"leader-auto-down",
|
||||||
"keep-oldest"
|
"keep-oldest"
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -48,7 +51,7 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
|||||||
builder.RequireThat(
|
builder.RequireThat(
|
||||||
!string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
|
!string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
|
||||||
&& AllowedStrategies.Contains(options.SplitBrainResolverStrategy),
|
&& AllowedStrategies.Contains(options.SplitBrainResolverStrategy),
|
||||||
$"ClusterOptions.SplitBrainResolverStrategy must be 'auto-down' or 'keep-oldest' for a " +
|
$"ClusterOptions.SplitBrainResolverStrategy must be 'auto-down', 'leader-auto-down' or 'keep-oldest' for a " +
|
||||||
$"two-node cluster; '{options.SplitBrainResolverStrategy}' would risk a total cluster " +
|
$"two-node cluster; '{options.SplitBrainResolverStrategy}' would risk a total cluster " +
|
||||||
"shutdown on a partition or an unreachability event.");
|
"shutdown on a partition or an unreachability event.");
|
||||||
|
|
||||||
@@ -70,7 +73,7 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
|||||||
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
|
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
|
||||||
"declared unreachable before a heartbeat can arrive.");
|
"declared unreachable before a heartbeat can arrive.");
|
||||||
|
|
||||||
// DownIfAlone is a keep-oldest knob; under auto-down each side downs the
|
// DownIfAlone is a keep-oldest knob; under (leader-)auto-down each side downs the
|
||||||
// unreachable peer regardless, so the flag is inert and any value is fine.
|
// unreachable peer regardless, so the flag is inert and any value is fine.
|
||||||
var isKeepOldest = string.Equals(
|
var isKeepOldest = string.Equals(
|
||||||
options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
|
options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
|
||||||
|
|||||||
@@ -235,7 +235,11 @@ public class AkkaHostedService : IHostedService
|
|||||||
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
|
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
|
||||||
/// downs the unreachable peer, so a crash of either node (oldest included) fails
|
/// downs the unreachable peer, so a crash of either node (oldest included) fails
|
||||||
/// over to the survivor; the accepted trade is dual-active during a real network
|
/// over to the survivor; the accepted trade is dual-active during a real network
|
||||||
/// partition. Any other value takes the SBR path, where the
|
/// partition. <c>leader-auto-down</c> (ScadaBridge#106, opt-in) installs the in-repo
|
||||||
|
/// <see cref="LeaderAutoDowningProvider"/> — the same semantics without Akka's deprecated
|
||||||
|
/// provider — with its duration at
|
||||||
|
/// <see cref="LeaderAutoDowningProvider.UnreachableAfterPath"/>.
|
||||||
|
/// Any other value takes the SBR path, where the
|
||||||
/// <c>keep-oldest down-if-alone</c> flag is emitted from
|
/// <c>keep-oldest down-if-alone</c> flag is emitted from
|
||||||
/// <see cref="ClusterOptions.DownIfAlone"/> rather than hard-coded, so the bound
|
/// <see cref="ClusterOptions.DownIfAlone"/> rather than hard-coded, so the bound
|
||||||
/// configuration value is actually consumed.
|
/// configuration value is actually consumed.
|
||||||
@@ -280,10 +284,17 @@ public class AkkaHostedService : IHostedService
|
|||||||
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
|
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
|
||||||
// the configured active-strategy (keep-oldest), which is partition-safe but
|
// the configured active-strategy (keep-oldest), which is partition-safe but
|
||||||
// cannot survive a crash of the oldest node in a two-node cluster.
|
// cannot survive a crash of the oldest node in a two-node cluster.
|
||||||
var downingBlock = string.Equals(
|
//
|
||||||
clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
|
// leader-auto-down (ScadaBridge#106, opt-in): the in-repo LeaderAutoDowningProvider — a
|
||||||
|
// port of AutoDowning with identical semantics that survives AutoDowning's eventual
|
||||||
|
// removal from Akka.NET. Its duration lives under our own key, so no
|
||||||
|
// auto-down-unreachable-after is emitted (which also silences Akka's deprecation warning).
|
||||||
|
var strategy = clusterOptions.SplitBrainResolverStrategy;
|
||||||
|
var downingBlock = string.Equals(strategy, "auto-down", StringComparison.OrdinalIgnoreCase)
|
||||||
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
|
? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
|
||||||
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
|
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
|
||||||
|
: string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase)
|
||||||
|
? $@"downing-provider-class = {QuoteHocon(LeaderAutoDowningProvider.ProviderClassName)}"
|
||||||
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
|
: $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
|
||||||
split-brain-resolver {{
|
split-brain-resolver {{
|
||||||
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
|
active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
|
||||||
@@ -293,8 +304,15 @@ public class AkkaHostedService : IHostedService
|
|||||||
}}
|
}}
|
||||||
}}";
|
}}";
|
||||||
|
|
||||||
|
// The leader-auto-down duration sits outside akka.cluster so Akka never mistakes it
|
||||||
|
// for one of its own (deprecated) auto-down keys. Only emitted for that strategy, so
|
||||||
|
// the auto-down / keep-oldest documents are byte-identical to before.
|
||||||
|
var leaderAutoDownBlock = string.Equals(strategy, "leader-auto-down", StringComparison.OrdinalIgnoreCase)
|
||||||
|
? $"{LeaderAutoDowningProvider.UnreachableAfterPath} = {DurationHocon(clusterOptions.StableAfter)}\n"
|
||||||
|
: string.Empty;
|
||||||
|
|
||||||
return $@"
|
return $@"
|
||||||
audit-telemetry-dispatcher {{
|
{leaderAutoDownBlock}audit-telemetry-dispatcher {{
|
||||||
type = ForkJoinDispatcher
|
type = ForkJoinDispatcher
|
||||||
throughput = 100
|
throughput = 100
|
||||||
dedicated-thread-pool {{
|
dedicated-thread-pool {{
|
||||||
|
|||||||
@@ -0,0 +1,318 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Cluster;
|
||||||
|
using Akka.Configuration;
|
||||||
|
using Akka.Event;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// In-repo replacement for Akka.NET's deprecated <c>Akka.Cluster.AutoDowning</c> provider
|
||||||
|
/// (ScadaBridge#51 plan, implemented by ScadaBridge#106). Selected by
|
||||||
|
/// <c>ScadaBridge:Cluster:SplitBrainResolverStrategy = "leader-auto-down"</c>; the HOCON is
|
||||||
|
/// emitted by <see cref="AkkaHostedService.BuildHocon"/>.
|
||||||
|
/// <para>
|
||||||
|
/// <b>Behaviour is a line-for-line port of Akka.NET 1.5.62 <c>AutoDowning</c> /
|
||||||
|
/// <c>AutoDown</c> / <c>AutoDownBase</c></b> (read from the decompiled 1.5.62 assembly), so
|
||||||
|
/// no operational contract changes when a pair switches from <c>auto-down</c>: the leader
|
||||||
|
/// among the REACHABLE members downs a peer that has stayed unreachable for
|
||||||
|
/// <c>unreachable-after</c> (= <c>ClusterOptions.StableAfter</c>); a non-leader records the
|
||||||
|
/// peer as pending and downs it if it later becomes leader; a peer that becomes reachable
|
||||||
|
/// again, or is removed, before the timer fires is forgotten. Because Akka computes the
|
||||||
|
/// leader over reachable members only, a crash of EITHER node of a pair (oldest included)
|
||||||
|
/// makes the survivor leader and it downs the crashed peer — the availability-first posture
|
||||||
|
/// of the 2026-07-21 decision, with dual-active on a real partition as the accepted trade.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Differences from <c>AutoDowning</c>, all deliberate and none behavioural:
|
||||||
|
/// the duration is read from <see cref="UnreachableAfterPath"/> rather than the deprecated
|
||||||
|
/// <c>akka.cluster.auto-down-unreachable-after</c> key (so Akka's boot-time deprecation
|
||||||
|
/// warning — "keep-majority will be used instead" — is no longer logged), and the
|
||||||
|
/// <see cref="DownRemovalMargin"/> is read from <c>akka.cluster.down-removal-margin</c>
|
||||||
|
/// directly because <c>ClusterSettings.DownRemovalMargin</c> is internal (same off-switch
|
||||||
|
/// parsing; default <c>off</c> = zero, exactly what AutoDowning returns).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// Akka instantiates this by reflection (<c>Akka.Cluster.DowningProvider.Load</c>) with the
|
||||||
|
/// <c>(ActorSystem, Cluster)</c> constructor; the downing actor is created as
|
||||||
|
/// <c>/system/cluster/core/daemon/downingProvider</c>.
|
||||||
|
/// </para>
|
||||||
|
/// </summary>
|
||||||
|
public sealed class LeaderAutoDowningProvider : IDowningProvider
|
||||||
|
{
|
||||||
|
/// <summary>HOCON path carrying the unreachable-before-down duration.</summary>
|
||||||
|
public const string UnreachableAfterPath = "scadabridge.cluster.leader-auto-down.unreachable-after";
|
||||||
|
|
||||||
|
/// <summary>Assembly-qualified type name for <c>akka.cluster.downing-provider-class</c>.</summary>
|
||||||
|
public static readonly string ProviderClassName =
|
||||||
|
$"{typeof(LeaderAutoDowningProvider).FullName}, {typeof(LeaderAutoDowningProvider).Assembly.GetName().Name}";
|
||||||
|
|
||||||
|
private readonly ActorSystem _system;
|
||||||
|
private readonly Akka.Cluster.Cluster _cluster;
|
||||||
|
|
||||||
|
/// <summary>Creates the provider; called by Akka via reflection.</summary>
|
||||||
|
/// <param name="system">The actor system the cluster runs in.</param>
|
||||||
|
/// <param name="cluster">The cluster extension being started.</param>
|
||||||
|
public LeaderAutoDowningProvider(ActorSystem system, Akka.Cluster.Cluster cluster)
|
||||||
|
{
|
||||||
|
_system = system;
|
||||||
|
_cluster = cluster;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public TimeSpan DownRemovalMargin
|
||||||
|
{
|
||||||
|
get
|
||||||
|
{
|
||||||
|
var clusterConfig = _system.Settings.Config.GetConfig("akka.cluster");
|
||||||
|
var raw = clusterConfig.GetString("down-removal-margin", string.Empty).ToLowerInvariant();
|
||||||
|
return raw is "off" or "false" or "no"
|
||||||
|
? TimeSpan.Zero
|
||||||
|
: clusterConfig.GetTimeSpan("down-removal-margin", null, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
/// <exception cref="ConfigurationException">
|
||||||
|
/// <see cref="UnreachableAfterPath"/> is missing or <c>off</c> — mirrors AutoDowning's
|
||||||
|
/// refusal to start without <c>auto-down-unreachable-after</c>.
|
||||||
|
/// </exception>
|
||||||
|
public Props DowningActorProps
|
||||||
|
{
|
||||||
|
get
|
||||||
|
{
|
||||||
|
var after = ReadUnreachableAfter(_system.Settings.Config);
|
||||||
|
if (!after.HasValue)
|
||||||
|
{
|
||||||
|
throw new ConfigurationException(
|
||||||
|
$"LeaderAutoDowningProvider selected but '{UnreachableAfterPath}' not set");
|
||||||
|
}
|
||||||
|
|
||||||
|
return LeaderAutoDown.Props(after.Value, _cluster);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Reads <see cref="UnreachableAfterPath"/> with Akka's off-switch semantics (Akka's own
|
||||||
|
/// <c>GetTimeSpanWithOffSwitch</c> is internal): missing or <c>off</c>/<c>false</c>/<c>no</c>
|
||||||
|
/// yields <see langword="null"/>.
|
||||||
|
/// </summary>
|
||||||
|
private static TimeSpan? ReadUnreachableAfter(Config config)
|
||||||
|
{
|
||||||
|
if (!config.HasPath(UnreachableAfterPath))
|
||||||
|
{
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
var raw = config.GetString(UnreachableAfterPath, string.Empty).ToLowerInvariant();
|
||||||
|
return raw is "off" or "false" or "no"
|
||||||
|
? null
|
||||||
|
: config.GetTimeSpan(UnreachableAfterPath, null, true);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The downing actor for <see cref="LeaderAutoDowningProvider"/> — port of Akka's internal
|
||||||
|
/// <c>AutoDown</c>: subscribes to cluster domain events and downs through
|
||||||
|
/// <see cref="Akka.Cluster.Cluster.Down"/>.
|
||||||
|
/// </summary>
|
||||||
|
internal sealed class LeaderAutoDown : LeaderAutoDownBase
|
||||||
|
{
|
||||||
|
private readonly Akka.Cluster.Cluster _cluster;
|
||||||
|
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||||
|
|
||||||
|
/// <summary>Props for the downing actor.</summary>
|
||||||
|
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||||
|
/// <param name="cluster">The cluster extension.</param>
|
||||||
|
public static Props Props(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster) =>
|
||||||
|
Akka.Actor.Props.Create(() => new LeaderAutoDown(unreachableAfter, cluster));
|
||||||
|
|
||||||
|
/// <summary>Creates the actor.</summary>
|
||||||
|
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||||
|
/// <param name="cluster">The cluster extension.</param>
|
||||||
|
public LeaderAutoDown(TimeSpan unreachableAfter, Akka.Cluster.Cluster cluster)
|
||||||
|
: base(unreachableAfter)
|
||||||
|
{
|
||||||
|
_cluster = cluster;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override Address SelfAddress => _cluster.SelfAddress;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
public override IScheduler Scheduler => Context.System.Scheduler;
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void PreStart()
|
||||||
|
{
|
||||||
|
_cluster.Subscribe(Self, typeof(ClusterEvent.IClusterDomainEvent));
|
||||||
|
base.PreStart();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void PostStop()
|
||||||
|
{
|
||||||
|
_cluster.Unsubscribe(Self);
|
||||||
|
base.PostStop();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
/// <exception cref="InvalidOperationException">Called on a non-leader (a logic error).</exception>
|
||||||
|
public override void Down(Address node)
|
||||||
|
{
|
||||||
|
if (!IsLeader)
|
||||||
|
{
|
||||||
|
throw new InvalidOperationException("Must be leader to down node");
|
||||||
|
}
|
||||||
|
|
||||||
|
_log.Info("Leader is auto-downing unreachable node [{0}]", node);
|
||||||
|
_cluster.Down(node);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The cluster-free decision logic of <see cref="LeaderAutoDown"/> — port of Akka's internal
|
||||||
|
/// <c>AutoDownBase</c>, split out (as Akka does) so the leader / reachability semantics can be
|
||||||
|
/// unit tested by feeding cluster events directly.
|
||||||
|
/// </summary>
|
||||||
|
internal abstract class LeaderAutoDownBase : UntypedActor
|
||||||
|
{
|
||||||
|
/// <summary>Timer message: <see cref="Node"/> has been unreachable for the full window.</summary>
|
||||||
|
/// <param name="Node">The member that was scheduled for downing.</param>
|
||||||
|
internal sealed record UnreachableTimeout(UniqueAddress Node);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Members in these states are not scheduled — Akka's
|
||||||
|
/// <c>MembershipState.ConvergenceSkipUnreachableWithMemberStatus</c> (Down, Exiting).
|
||||||
|
/// </summary>
|
||||||
|
private static readonly ImmutableHashSet<MemberStatus> SkipMemberStatus =
|
||||||
|
ImmutableHashSet.Create(MemberStatus.Down, MemberStatus.Exiting);
|
||||||
|
|
||||||
|
private readonly TimeSpan _unreachableAfter;
|
||||||
|
private ImmutableDictionary<UniqueAddress, ICancelable> _scheduledUnreachable =
|
||||||
|
ImmutableDictionary<UniqueAddress, ICancelable>.Empty;
|
||||||
|
private ImmutableHashSet<UniqueAddress> _pendingUnreachable = ImmutableHashSet<UniqueAddress>.Empty;
|
||||||
|
|
||||||
|
/// <summary>Creates the base logic.</summary>
|
||||||
|
/// <param name="unreachableAfter">How long a member must stay unreachable before it is downed.</param>
|
||||||
|
protected LeaderAutoDownBase(TimeSpan unreachableAfter)
|
||||||
|
{
|
||||||
|
_unreachableAfter = unreachableAfter;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Whether this node is currently the cluster leader.</summary>
|
||||||
|
protected bool IsLeader { get; private set; }
|
||||||
|
|
||||||
|
/// <summary>This node's address, compared against the leader address.</summary>
|
||||||
|
public abstract Address SelfAddress { get; }
|
||||||
|
|
||||||
|
/// <summary>The scheduler used for the unreachable timers.</summary>
|
||||||
|
public abstract IScheduler Scheduler { get; }
|
||||||
|
|
||||||
|
/// <summary>Downs <paramref name="node"/>; only ever called while leader.</summary>
|
||||||
|
/// <param name="node">The member to down.</param>
|
||||||
|
public abstract void Down(Address node);
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void PostStop()
|
||||||
|
{
|
||||||
|
foreach (var cancelable in _scheduledUnreachable.Values)
|
||||||
|
{
|
||||||
|
cancelable.Cancel();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <inheritdoc />
|
||||||
|
protected override void OnReceive(object message)
|
||||||
|
{
|
||||||
|
switch (message)
|
||||||
|
{
|
||||||
|
case ClusterEvent.CurrentClusterState state:
|
||||||
|
IsLeader = state.Leader != null && state.Leader.Equals(SelfAddress);
|
||||||
|
foreach (var member in state.Unreachable)
|
||||||
|
{
|
||||||
|
UnreachableMember(member);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case ClusterEvent.UnreachableMember unreachable:
|
||||||
|
UnreachableMember(unreachable.Member);
|
||||||
|
break;
|
||||||
|
|
||||||
|
case ClusterEvent.ReachableMember reachable:
|
||||||
|
Remove(reachable.Member.UniqueAddress);
|
||||||
|
break;
|
||||||
|
|
||||||
|
case ClusterEvent.MemberRemoved removed:
|
||||||
|
Remove(removed.Member.UniqueAddress);
|
||||||
|
break;
|
||||||
|
|
||||||
|
case ClusterEvent.LeaderChanged leaderChanged:
|
||||||
|
IsLeader = leaderChanged.Leader != null && leaderChanged.Leader.Equals(SelfAddress);
|
||||||
|
if (IsLeader)
|
||||||
|
{
|
||||||
|
foreach (var node in _pendingUnreachable)
|
||||||
|
{
|
||||||
|
Down(node.Address);
|
||||||
|
}
|
||||||
|
|
||||||
|
_pendingUnreachable = ImmutableHashSet<UniqueAddress>.Empty;
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
case UnreachableTimeout timeout:
|
||||||
|
if (_scheduledUnreachable.ContainsKey(timeout.Node))
|
||||||
|
{
|
||||||
|
_scheduledUnreachable = _scheduledUnreachable.Remove(timeout.Node);
|
||||||
|
DownOrAddPending(timeout.Node);
|
||||||
|
}
|
||||||
|
break;
|
||||||
|
|
||||||
|
// Every other IClusterDomainEvent is ignored, exactly as AutoDownBase does.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void UnreachableMember(Member member)
|
||||||
|
{
|
||||||
|
if (!SkipMemberStatus.Contains(member.Status) && !_scheduledUnreachable.ContainsKey(member.UniqueAddress))
|
||||||
|
{
|
||||||
|
ScheduleUnreachable(member.UniqueAddress);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void ScheduleUnreachable(UniqueAddress node)
|
||||||
|
{
|
||||||
|
if (_unreachableAfter == TimeSpan.Zero)
|
||||||
|
{
|
||||||
|
DownOrAddPending(node);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
var cancelable = Scheduler.ScheduleTellOnceCancelable(
|
||||||
|
_unreachableAfter, Self, new UnreachableTimeout(node), Self);
|
||||||
|
_scheduledUnreachable = _scheduledUnreachable.Add(node, cancelable);
|
||||||
|
}
|
||||||
|
|
||||||
|
private void DownOrAddPending(UniqueAddress node)
|
||||||
|
{
|
||||||
|
if (IsLeader)
|
||||||
|
{
|
||||||
|
Down(node.Address);
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
_pendingUnreachable = _pendingUnreachable.Add(node);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
private void Remove(UniqueAddress node)
|
||||||
|
{
|
||||||
|
if (_scheduledUnreachable.TryGetValue(node, out var cancelable))
|
||||||
|
{
|
||||||
|
cancelable.Cancel();
|
||||||
|
}
|
||||||
|
|
||||||
|
_scheduledUnreachable = _scheduledUnreachable.Remove(node);
|
||||||
|
_pendingUnreachable = _pendingUnreachable.Remove(node);
|
||||||
|
}
|
||||||
|
}
|
||||||
+29
@@ -298,4 +298,33 @@ public class ClusterOptionsValidatorTests
|
|||||||
Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage);
|
Assert.Contains("PartnerProbeIntervalMs", result.FailureMessage);
|
||||||
Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage);
|
Assert.Contains("ProbeConnectTimeoutMs", result.FailureMessage);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that the leader-auto-down strategy (ScadaBridge#106) passes validation, case-insensitively.</summary>
|
||||||
|
/// <param name="strategy">The strategy spelling.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData("leader-auto-down")]
|
||||||
|
[InlineData("Leader-Auto-Down")]
|
||||||
|
public void LeaderAutoDownStrategy_Passes(string strategy)
|
||||||
|
{
|
||||||
|
var options = ValidOptions();
|
||||||
|
options.SplitBrainResolverStrategy = strategy;
|
||||||
|
options.DownIfAlone = false; // keep-oldest-only knob; inert here, must not block startup
|
||||||
|
|
||||||
|
var result = new ClusterOptionsValidator().Validate(null, options);
|
||||||
|
|
||||||
|
Assert.True(result.Succeeded, result.FailureMessage);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that the rejection message lists leader-auto-down among the supported strategies.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void UnsupportedStrategy_MessageNamesLeaderAutoDown()
|
||||||
|
{
|
||||||
|
var options = ValidOptions();
|
||||||
|
options.SplitBrainResolverStrategy = "leader-autodown";
|
||||||
|
|
||||||
|
var result = new ClusterOptionsValidator().Validate(null, options);
|
||||||
|
|
||||||
|
Assert.True(result.Failed);
|
||||||
|
Assert.Contains("'leader-auto-down'", result.FailureMessage);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -250,4 +250,76 @@ public class HoconBuilderTests
|
|||||||
Assert.Equal(1, config.GetInt("akka.cluster.min-nr-of-members"));
|
Assert.Equal(1, config.GetInt("akka.cluster.min-nr-of-members"));
|
||||||
Assert.True(config.GetBoolean("akka.cluster.run-coordinated-shutdown-when-down"));
|
Assert.True(config.GetBoolean("akka.cluster.run-coordinated-shutdown-when-down"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies that the leader-auto-down strategy (ScadaBridge#106) installs the in-repo
|
||||||
|
/// provider, carries StableAfter under its own key, and emits neither Akka's deprecated
|
||||||
|
/// auto-down key nor an SBR section.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void BuildHocon_LeaderAutoDownStrategy_EmitsInRepoProvider_AndNoDeprecatedKey()
|
||||||
|
{
|
||||||
|
var cluster = DefaultCluster();
|
||||||
|
cluster.SplitBrainResolverStrategy = "leader-auto-down";
|
||||||
|
cluster.StableAfter = TimeSpan.FromMilliseconds(15250);
|
||||||
|
|
||||||
|
var hocon = AkkaHostedService.BuildHocon(
|
||||||
|
DefaultNode(), cluster, new[] { "Central" },
|
||||||
|
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));
|
||||||
|
|
||||||
|
var config = ConfigurationFactory.ParseString(hocon);
|
||||||
|
Assert.Equal(
|
||||||
|
"ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host",
|
||||||
|
config.GetString("akka.cluster.downing-provider-class"));
|
||||||
|
// The emitted name must resolve the way Akka resolves it (Type.GetType(name, true)).
|
||||||
|
Assert.Equal(typeof(LeaderAutoDowningProvider),
|
||||||
|
Type.GetType(config.GetString("akka.cluster.downing-provider-class"), throwOnError: true));
|
||||||
|
Assert.Equal(
|
||||||
|
TimeSpan.FromMilliseconds(15250),
|
||||||
|
config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath));
|
||||||
|
// No deprecated key (that is what triggers Akka's "keep-majority will be used" warning)
|
||||||
|
// and no SBR section alongside the provider.
|
||||||
|
Assert.False(config.HasPath("akka.cluster.auto-down-unreachable-after"));
|
||||||
|
Assert.False(config.HasPath("akka.cluster.split-brain-resolver.active-strategy"));
|
||||||
|
// Document stays intact after the downing block.
|
||||||
|
Assert.Equal(1, config.GetInt("akka.cluster.min-nr-of-members"));
|
||||||
|
Assert.True(config.GetBoolean("akka.cluster.run-coordinated-shutdown-when-down"));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Verifies that the leader-auto-down strategy name is case-insensitive.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void BuildHocon_LeaderAutoDownStrategy_IsCaseInsensitive()
|
||||||
|
{
|
||||||
|
var cluster = DefaultCluster();
|
||||||
|
cluster.SplitBrainResolverStrategy = "Leader-Auto-Down";
|
||||||
|
|
||||||
|
var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
|
||||||
|
DefaultNode(), cluster, new[] { "Central" },
|
||||||
|
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
|
||||||
|
|
||||||
|
Assert.Equal(LeaderAutoDowningProvider.ProviderClassName,
|
||||||
|
config.GetString("akka.cluster.downing-provider-class"));
|
||||||
|
Assert.Equal(TimeSpan.FromSeconds(15), config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Verifies the leader-auto-down key is emitted ONLY for that strategy: the existing
|
||||||
|
/// auto-down and keep-oldest documents are unchanged by ScadaBridge#106.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="strategy">An existing strategy.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData("auto-down")]
|
||||||
|
[InlineData("keep-oldest")]
|
||||||
|
public void BuildHocon_OtherStrategies_DoNotEmitLeaderAutoDownKey(string strategy)
|
||||||
|
{
|
||||||
|
var cluster = DefaultCluster();
|
||||||
|
cluster.SplitBrainResolverStrategy = strategy;
|
||||||
|
|
||||||
|
var hocon = AkkaHostedService.BuildHocon(
|
||||||
|
DefaultNode(), cluster, new[] { "Central" },
|
||||||
|
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15));
|
||||||
|
|
||||||
|
Assert.DoesNotContain("leader-auto-down", hocon);
|
||||||
|
Assert.DoesNotContain(nameof(LeaderAutoDowningProvider), hocon);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,215 @@
|
|||||||
|
using System.Collections.Immutable;
|
||||||
|
using System.Reflection;
|
||||||
|
using Akka.Actor;
|
||||||
|
using Akka.Cluster;
|
||||||
|
using Akka.TestKit.Xunit2;
|
||||||
|
using Akka.Util;
|
||||||
|
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.ScadaBridge.Host.Tests;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// ScadaBridge#106: the decision logic of <see cref="LeaderAutoDownBase"/> must match Akka.NET
|
||||||
|
/// 1.5.62 <c>AutoDownBase</c> exactly. These cases are the ones Akka's own <c>AutoDownSpec</c>
|
||||||
|
/// pins — cluster events are fed straight to the actor and the <c>Down</c> seam reports to a
|
||||||
|
/// probe, so the leader-only and reachability semantics are proven without a running cluster.
|
||||||
|
/// The two-node real-cluster behaviour is proven separately in IntegrationTests
|
||||||
|
/// (<c>LeaderAutoDownFailoverTests</c>).
|
||||||
|
/// </summary>
|
||||||
|
public class LeaderAutoDownTests : TestKit
|
||||||
|
{
|
||||||
|
private static readonly Address MemberA = new("akka.tcp", "sys", "a", 2552);
|
||||||
|
private static readonly Address MemberB = new("akka.tcp", "sys", "b", 2552);
|
||||||
|
private static readonly Address MemberC = new("akka.tcp", "sys", "c", 2552);
|
||||||
|
|
||||||
|
private sealed record DownCalled(Address Address);
|
||||||
|
|
||||||
|
/// <summary>Test double: <c>Down</c> reports to a probe instead of calling the cluster.</summary>
|
||||||
|
private sealed class TestLeaderAutoDown : LeaderAutoDownBase
|
||||||
|
{
|
||||||
|
private readonly IActorRef _probe;
|
||||||
|
|
||||||
|
public TestLeaderAutoDown(Address selfAddress, TimeSpan unreachableAfter, IActorRef probe)
|
||||||
|
: base(unreachableAfter)
|
||||||
|
{
|
||||||
|
SelfAddress = selfAddress;
|
||||||
|
_probe = probe;
|
||||||
|
}
|
||||||
|
|
||||||
|
public override Address SelfAddress { get; }
|
||||||
|
|
||||||
|
public override IScheduler Scheduler => Context.System.Scheduler;
|
||||||
|
|
||||||
|
public override void Down(Address node)
|
||||||
|
{
|
||||||
|
// Report EVERY call, leader or not: a base that wrongly downs while not leader must
|
||||||
|
// show up as an unexpected message, not be swallowed by a throw-and-restart.
|
||||||
|
_probe.Tell(new DownCalled(node));
|
||||||
|
if (!IsLeader)
|
||||||
|
{
|
||||||
|
_probe.Tell("down-called-while-not-leader");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Akka's <c>Member</c> constructors are internal; tests build members via the internal factory.</summary>
|
||||||
|
private static Member CreateMember(Address address, MemberStatus status)
|
||||||
|
{
|
||||||
|
var create = typeof(Member).GetMethod(
|
||||||
|
"Create",
|
||||||
|
BindingFlags.Static | BindingFlags.NonPublic,
|
||||||
|
new[] { typeof(UniqueAddress), typeof(int), typeof(MemberStatus), typeof(ImmutableHashSet<string>), typeof(AppVersion) })
|
||||||
|
?? throw new InvalidOperationException("Akka.Cluster.Member.Create(…) not found — Akka internals changed.");
|
||||||
|
return (Member)create.Invoke(null, new object[]
|
||||||
|
{
|
||||||
|
new UniqueAddress(address, 1), 1, status, ImmutableHashSet<string>.Empty, AppVersion.Zero,
|
||||||
|
})!;
|
||||||
|
}
|
||||||
|
|
||||||
|
private IActorRef AutoDownActor(TimeSpan unreachableAfter) =>
|
||||||
|
Sys.ActorOf(Props.Create(() => new TestLeaderAutoDown(MemberA, unreachableAfter, TestActor)));
|
||||||
|
|
||||||
|
/// <summary>A non-leader never downs an unreachable member.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void NotLeader_DoesNotDownUnreachable()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The leader downs an unreachable member immediately when the window is zero.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Leader_DownsUnreachable_WhenWindowIsZero()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up)));
|
||||||
|
ExpectMsg(new DownCalled(MemberB));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The leader downs an unreachable member only after the full window.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Leader_DownsUnreachable_AfterWindow()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(2));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, MemberStatus.Up)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1));
|
||||||
|
ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The initial snapshot drives leadership and schedules members already unreachable.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void CurrentClusterState_Leader_DownsAlreadyUnreachable()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||||
|
a.Tell(new ClusterEvent.CurrentClusterState(
|
||||||
|
ImmutableSortedSet<Member>.Empty,
|
||||||
|
ImmutableHashSet.Create(b),
|
||||||
|
ImmutableHashSet<Address>.Empty,
|
||||||
|
MemberA,
|
||||||
|
ImmutableDictionary<string, Address>.Empty));
|
||||||
|
ExpectMsg(new DownCalled(MemberB));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A pending member is downed as soon as this node becomes leader.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void BecomingLeader_DownsPendingUnreachable()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
ExpectMsg(new DownCalled(MemberC));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A member that timed out while not leader is downed on becoming leader.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void BecomingLeader_AfterWindow_DownsPendingUnreachable()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(1));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(2));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
ExpectMsg(new DownCalled(MemberC));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Losing leadership inside the window means no Down when the timer fires.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void LosingLeadership_InsideWindow_DoesNotDown()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(2));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberC, MemberStatus.Up)));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A member that becomes reachable again inside the window is not downed.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void ReachableAgain_InsideWindow_DoesNotDown()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(2));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||||
|
a.Tell(new ClusterEvent.ReachableMember(b));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A pending member that becomes reachable again is not downed on becoming leader.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void ReachableAgain_WhilePending_IsForgotten()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberB));
|
||||||
|
var c = CreateMember(MemberC, MemberStatus.Up);
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(c));
|
||||||
|
a.Tell(new ClusterEvent.ReachableMember(c));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A member removed inside the window is not downed.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void RemovedInsideWindow_DoesNotDown()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(2));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||||
|
a.Tell(new ClusterEvent.MemberRemoved(b.Copy(MemberStatus.Removed), MemberStatus.Up));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(3));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Members already Down or Exiting are skipped (Akka's ConvergenceSkipUnreachableWithMemberStatus).</summary>
|
||||||
|
/// <param name="status">The member status.</param>
|
||||||
|
[Theory]
|
||||||
|
[InlineData(MemberStatus.Down)]
|
||||||
|
[InlineData(MemberStatus.Exiting)]
|
||||||
|
public void AlreadyDownOrExiting_IsNotDownedAgain(MemberStatus status)
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.Zero);
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(CreateMember(MemberB, status)));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(1));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>A repeated UnreachableMember for an already-scheduled member does not schedule a second Down.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void RepeatedUnreachable_DownsOnce()
|
||||||
|
{
|
||||||
|
var a = AutoDownActor(TimeSpan.FromSeconds(1));
|
||||||
|
a.Tell(new ClusterEvent.LeaderChanged(MemberA));
|
||||||
|
var b = CreateMember(MemberB, MemberStatus.Up);
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||||
|
a.Tell(new ClusterEvent.UnreachableMember(b));
|
||||||
|
ExpectMsg(new DownCalled(MemberB), TimeSpan.FromSeconds(3));
|
||||||
|
ExpectNoMsg(TimeSpan.FromSeconds(2));
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,123 @@
|
|||||||
|
using Akka.Actor;
|
||||||
|
using Xunit;
|
||||||
|
using ZB.MOM.WW.ScadaBridge.Host.Actors;
|
||||||
|
|
||||||
|
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// ScadaBridge#106: the in-repo <see cref="LeaderAutoDowningProvider"/> (strategy
|
||||||
|
/// <c>leader-auto-down</c>) must give a REAL two-node cluster exactly the outcomes Akka's
|
||||||
|
/// <c>AutoDowning</c> (<c>auto-down</c>) gives in <see cref="SbrFailoverTests"/>: a hard crash
|
||||||
|
/// of EITHER node — the oldest included — is downed and removed by the survivor and the
|
||||||
|
/// singleton ends up answering on the survivor. The two failover cases are clones of
|
||||||
|
/// <c>SbrFailoverTests.AutoDown_HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton</c> and
|
||||||
|
/// <c>…AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton</c>, differing only in the strategy.
|
||||||
|
/// Under a provider that never downs, the member-removal step times out (the crashed member
|
||||||
|
/// stays Unreachable forever) — that is what gives them teeth.
|
||||||
|
/// </summary>
|
||||||
|
public class LeaderAutoDownFailoverTests
|
||||||
|
{
|
||||||
|
private const string Strategy = "leader-auto-down";
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The configured provider is the one Akka actually instantiated and started: the Cluster
|
||||||
|
/// extension holds a <see cref="LeaderAutoDowningProvider"/> and its downing actor is running
|
||||||
|
/// under the cluster daemon on both nodes.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task LeaderAutoDown_ProviderIsWhatAkkaLoads_OnBothNodes()
|
||||||
|
{
|
||||||
|
await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy);
|
||||||
|
|
||||||
|
foreach (var node in new[] { cluster.NodeA, cluster.NodeB })
|
||||||
|
{
|
||||||
|
var akkaCluster = Akka.Cluster.Cluster.Get(node);
|
||||||
|
Assert.IsType<LeaderAutoDowningProvider>(akkaCluster.DowningProvider);
|
||||||
|
|
||||||
|
// ClusterCoreDaemon creates the provider's DowningActorProps as "downingProvider".
|
||||||
|
var identity = await node.ActorSelection("/system/cluster/core/daemon/downingProvider")
|
||||||
|
.Ask<ActorIdentity>(new Identify("downing"), TimeSpan.FromSeconds(5));
|
||||||
|
Assert.NotNull(identity.Subject);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// With leader-auto-down, a hard crash of the OLDEST node lets the younger survivor down it
|
||||||
|
/// and take over the singleton — the direction two-node keep-oldest cannot survive.
|
||||||
|
/// </summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task LeaderAutoDown_HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton()
|
||||||
|
{
|
||||||
|
await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy);
|
||||||
|
SbrFailoverTests.StartSingleton(cluster.NodeA); // oldest hosts the singleton initially
|
||||||
|
var (_, proxyB) = SbrFailoverTests.StartSingleton(cluster.NodeB);
|
||||||
|
|
||||||
|
var echo = await proxyB.Ask<string>("ping", TimeSpan.FromSeconds(20));
|
||||||
|
Assert.Equal("ping", echo);
|
||||||
|
|
||||||
|
var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeA).SelfAddress;
|
||||||
|
await TwoNodeClusterFixture.CrashNode(cluster.NodeA);
|
||||||
|
|
||||||
|
// 1) The younger survivor must DOWN and REMOVE the crashed OLDEST member.
|
||||||
|
await TwoNodeClusterFixture.WaitForMemberRemoved(
|
||||||
|
cluster.NodeB, victimAddress, TimeSpan.FromSeconds(30));
|
||||||
|
|
||||||
|
// 2) B must still be a functioning cluster member (not self-downed) …
|
||||||
|
var clusterB = Akka.Cluster.Cluster.Get(cluster.NodeB);
|
||||||
|
Assert.False(clusterB.IsTerminated, "survivor's Cluster extension terminated — it downed itself");
|
||||||
|
|
||||||
|
// 3) … and the singleton must migrate to B and answer again.
|
||||||
|
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||||
|
Exception? last = null;
|
||||||
|
while (DateTime.UtcNow < deadline)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var echo2 = await proxyB.Ask<string>("ping-after-oldest-crash", TimeSpan.FromSeconds(3));
|
||||||
|
Assert.Equal("ping-after-oldest-crash", echo2);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
catch (Exception ex) { last = ex; }
|
||||||
|
}
|
||||||
|
throw new Xunit.Sdk.XunitException(
|
||||||
|
$"Singleton never migrated to the younger survivor after the oldest crashed under leader-auto-down: {last}");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>With leader-auto-down, a hard crash of the younger node leaves the singleton on the oldest node.</summary>
|
||||||
|
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||||
|
[Fact]
|
||||||
|
public async Task LeaderAutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton()
|
||||||
|
{
|
||||||
|
await using var cluster = await TwoNodeClusterFixture.StartAsync(strategy: Strategy);
|
||||||
|
var (_, proxyA) = SbrFailoverTests.StartSingleton(cluster.NodeA);
|
||||||
|
SbrFailoverTests.StartSingleton(cluster.NodeB);
|
||||||
|
|
||||||
|
Assert.Equal("ping", await proxyA.Ask<string>("ping", TimeSpan.FromSeconds(20)));
|
||||||
|
|
||||||
|
var victimAddress = Akka.Cluster.Cluster.Get(cluster.NodeB).SelfAddress;
|
||||||
|
await TwoNodeClusterFixture.CrashNode(cluster.NodeB);
|
||||||
|
|
||||||
|
await TwoNodeClusterFixture.WaitForMemberRemoved(
|
||||||
|
cluster.NodeA, victimAddress, TimeSpan.FromSeconds(30));
|
||||||
|
|
||||||
|
Assert.False(Akka.Cluster.Cluster.Get(cluster.NodeA).IsTerminated,
|
||||||
|
"surviving oldest's Cluster extension terminated — it downed itself");
|
||||||
|
|
||||||
|
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||||
|
Exception? last = null;
|
||||||
|
while (DateTime.UtcNow < deadline)
|
||||||
|
{
|
||||||
|
try
|
||||||
|
{
|
||||||
|
var echo2 = await proxyA.Ask<string>("ping-after-crash", TimeSpan.FromSeconds(3));
|
||||||
|
Assert.Equal("ping-after-crash", echo2);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
catch (Exception ex) { last = ex; }
|
||||||
|
}
|
||||||
|
throw new Xunit.Sdk.XunitException(
|
||||||
|
$"Singleton stopped answering on the surviving oldest node under leader-auto-down: {last}");
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -34,7 +34,7 @@ public class SbrFailoverTests
|
|||||||
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
|
public EchoActor() => ReceiveAny(msg => Sender.Tell(msg));
|
||||||
}
|
}
|
||||||
|
|
||||||
private static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys)
|
internal static (IActorRef manager, IActorRef proxy) StartSingleton(ActorSystem sys)
|
||||||
{
|
{
|
||||||
var manager = sys.ActorOf(ClusterSingletonManager.Props(
|
var manager = sys.ActorOf(ClusterSingletonManager.Props(
|
||||||
Props.Create(() => new EchoActor()),
|
Props.Create(() => new EchoActor()),
|
||||||
|
|||||||
Reference in New Issue
Block a user