feat(cluster): site-pair manual failover relayed from the central UI (Task 10)
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
This commit is contained in:
@@ -0,0 +1,56 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
|
||||
|
||||
/// <summary>
|
||||
/// Deliberate, operator-initiated failover of a two-node cluster: the current active member
|
||||
/// leaves GRACEFULLY so <c>ClusterSingletonManager</c> hands its singletons to the survivor.
|
||||
///
|
||||
/// <para>Lives beside <see cref="ActiveNodeEvaluator"/> — the two share one definition of
|
||||
/// "active" (oldest Up member in a role scope), and they must not drift: the node asked to
|
||||
/// leave has to be exactly the node hosting the singletons. Placed in Communication rather
|
||||
/// than Host because BOTH sides need it — the central pair fails over via the Host's
|
||||
/// <c>AkkaManualFailoverService</c>, and a site pair fails over inside
|
||||
/// <c>SiteCommunicationActor</c>, which cannot reference Host.</para>
|
||||
///
|
||||
/// <para><b>Leave, never Down.</b> A Down skips singleton hand-off and hands the outcome to
|
||||
/// the downing strategy — the wrong tool for a planned role swap.</para>
|
||||
/// </summary>
|
||||
public static class ClusterFailoverCoordinator
|
||||
{
|
||||
/// <summary>
|
||||
/// Asks the oldest Up member carrying <paramref name="role"/> to leave the cluster.
|
||||
/// Mirrors <see cref="ActiveNodeEvaluator.SelfIsOldestUp"/>'s rule, so the node acted on is
|
||||
/// the singleton host — never Akka's cluster <i>leader</i>, whose address-ordered definition
|
||||
/// diverges from singleton placement once the original first node restarts and rejoins.
|
||||
/// </summary>
|
||||
/// <param name="system">Actor system whose cluster is acted on.</param>
|
||||
/// <param name="role">Role scope. Central uses <c>Central</c>; a site pair uses its
|
||||
/// site-specific <c>site-{SiteId}</c> role, since site singletons are scoped to that role.</param>
|
||||
/// <param name="dryRun">Resolve and return the target WITHOUT issuing the Leave. Used to
|
||||
/// name the target in an audit row before acting, and to answer "would this work?".</param>
|
||||
/// <returns>The address that leaves (or would leave), or <c>null</c> when fewer than two Up
|
||||
/// members carry the role — failing over a lone node is an outage, not a failover.</returns>
|
||||
public static Address? FailOverOldest(ActorSystem system, string role, bool dryRun = false)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var withRole = cluster.State.Members
|
||||
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
|
||||
.OrderBy(m => m, Member.AgeOrdering)
|
||||
.ToList();
|
||||
|
||||
if (withRole.Count < 2)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
var oldest = withRole[0];
|
||||
if (!dryRun)
|
||||
{
|
||||
cluster.Leave(oldest.Address);
|
||||
}
|
||||
|
||||
return oldest.Address;
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user