Joseph Doherty
c8e2f4da02
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
52 lines
2.5 KiB
C#
52 lines
2.5 KiB
C#
using Akka.Actor;
|
|
using Akka.Cluster;
|
|
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
|
|
|
|
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
|
|
|
/// <summary>
|
|
/// Real-cluster proof for the central→site failover relay (Task 10). The unit tests in
|
|
/// <c>SiteCommunicationActorTests</c> pin the routing and guard logic with the failover action
|
|
/// stubbed out; this pins the part they cannot — that the shared
|
|
/// <see cref="ClusterFailoverCoordinator"/> actually moves a SITE pair when scoped to the
|
|
/// site-specific role.
|
|
///
|
|
/// <para>The site-specific role scope is the load-bearing detail: site singletons (the
|
|
/// Deployment Manager) are placed on <c>site-{SiteId}</c>, not on the base <c>Site</c> role, so
|
|
/// failing over the wrong scope would move the wrong node.</para>
|
|
/// </summary>
|
|
public sealed class SiteFailoverRelayTests
|
|
{
|
|
[Fact]
|
|
public async Task Failing_over_a_site_pair_moves_the_oldest_and_the_survivor_takes_over()
|
|
{
|
|
// A site pair, both nodes carrying the site-specific role.
|
|
await using var f = await TwoNodeClusterFixture.StartAsync(role: "site-SiteA");
|
|
var oldest = Akka.Cluster.Cluster.Get(f.NodeA);
|
|
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(oldest, "site-SiteA"),
|
|
"precondition: NodeA must be the active (oldest Up) site node");
|
|
|
|
// Issued from the other node, as the relay does when contact rotation lands there.
|
|
var target = ClusterFailoverCoordinator.FailOverOldest(f.NodeB, "site-SiteA");
|
|
|
|
Assert.Equal(oldest.SelfAddress, target);
|
|
await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
|
|
await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
|
|
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB), "site-SiteA"));
|
|
}
|
|
|
|
[Fact]
|
|
public async Task A_site_role_scope_that_matches_no_member_is_refused()
|
|
{
|
|
// Guards the role-scoping mistake directly: asking for a site that isn't this pair
|
|
// must find no members and refuse, rather than falling back to some other node.
|
|
await using var f = await TwoNodeClusterFixture.StartAsync(role: "site-SiteA");
|
|
|
|
var target = ClusterFailoverCoordinator.FailOverOldest(f.NodeB, "site-SiteB");
|
|
|
|
Assert.Null(target);
|
|
// Positive control: the pair is untouched and still fully formed.
|
|
await TwoNodeClusterFixture.WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(10));
|
|
}
|
|
}
|