dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(cluster): site-pair manual failover relayed from the central UI (Task 10)

Browse Source 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
This commit is contained in:
Joseph Doherty
2026-07-22 07:48:56 -04:00
parent d66e0d585f
commit c8e2f4da02
12 changed files with 621 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/HealthFailoverButtonTests.cs
+85
View File
@@ -153,4 +153,89 @@ public class HealthFailoverButtonTests : BunitContext
Assert.Contains("no standby", cut.Markup, StringComparison.OrdinalIgnoreCase);
}
// ---- Site-pair failover (Task 10) ------------------------------------------
// Same control, SiteId set. The differences that matter: it calls the site path,
// and its confirmation must NOT claim the admin's own page will drop (it won't —
// the site is a different cluster).
private IRenderedComponent<CentralFailoverControl> RenderForSite(int onlineNodes, string siteId)
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<CentralFailoverControl>(0);
builder.AddAttribute(1, nameof(CentralFailoverControl.OnlineCentralNodeCount), onlineNodes);
builder.AddAttribute(2, nameof(CentralFailoverControl.SiteId), siteId);
builder.CloseComponent();
})));
return host.FindComponent<CentralFailoverControl>();
}
[Fact]
public async Task Site_card_confirming_triggers_the_site_failover_path()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "akka.tcp://scadabridge@site-a-a:8082", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _failover.Received(1).FailOverSiteAsync("SiteA", "tester");
await _failover.DidNotReceive().FailOverCentralAsync(Arg.Any<string>());
}
[Fact]
public async Task Site_card_confirmation_does_not_claim_this_page_disconnects()
{
// Central failover drops the admin's circuit; a SITE failover does not, and saying
// otherwise would train operators to distrust the warning that does matter.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(true, "addr", null)));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
await _dialog.Received(1).ConfirmAsync(
Arg.Any<string>(),
Arg.Is<string>(m => !m.Contains("reconnect", StringComparison.OrdinalIgnoreCase)),
Arg.Any<bool>());
}
[Fact]
public async Task Site_card_surfaces_a_refusal_from_the_site()
{
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "No standby available — failing over a lone node would be an outage.")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("No standby available", cut.Markup);
}
[Fact]
public async Task Site_card_surfaces_an_unreachable_site_distinctly_from_a_refusal()
{
// A timeout is not a "no" from the site — the operator must be able to tell the
// difference, because the failover may or may not have taken effect.
Arrange("Administrator");
_dialog.ConfirmAsync(Arg.Any<string>(), Arg.Any<string>(), Arg.Any<bool>()).Returns(true);
_failover.FailOverSiteAsync(Arg.Any<string>(), Arg.Any<string>())
.Returns(Task.FromResult(new SiteFailoverOutcome(
false, null, "Site did not respond: Ask timed out")));
var cut = RenderForSite(onlineNodes: 2, siteId: "SiteA");
await cut.Find("button").ClickAsync(new());
Assert.Contains("did not respond", cut.Markup);
}
}
tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Pages/HealthPageTests.cs
+45 -11
View File
@@ -2,11 +2,14 @@ using System.Security.Claims;
using ZB.MOM.WW.ScadaBridge.Security;
using Akka.Actor;
using Bunit;
using Microsoft.AspNetCore.Authorization;
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.Authorization;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
using Microsoft.Extensions.Options;
using NSubstitute;
using ZB.MOM.WW.ScadaBridge.CentralUI.Components.Shared;
using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
@@ -105,12 +108,43 @@ public class HealthPageTests : BunitContext
var user = new ClaimsPrincipal(new ClaimsIdentity(claims, "TestAuth"));
Services.AddSingleton<AuthenticationStateProvider>(new TestAuthStateProvider(user));
Services.AddAuthorizationCore();
// The failover control's AuthorizeView evaluates the named RequireAdmin policy, so the
// real policy set has to be registered or the page throws on any card that renders it.
AuthorizationPolicies.AddScadaBridgeAuthorization(Services);
// BunitContext pre-registers a placeholder IAuthorizationService that throws when a
// policy is evaluated; force the real one (same note as NavMenuTests).
Services.AddSingleton<IAuthorizationService, DefaultAuthorizationService>();
// The cluster cards embed CentralFailoverControl (Task 10), which injects these two.
// The page's DI graph genuinely requires them in production, so the test supplies
// them rather than the component making its dependencies optional. Behaviour of the
// control itself is covered by HealthFailoverButtonTests.
Services.AddSingleton(Substitute.For<IManualFailoverService>());
Services.AddSingleton(Substitute.For<IDialogService>());
}
/// <summary>
/// Renders the dashboard the way the app does — inside a
/// <see cref="CascadingAuthenticationState"/>, which the real layout supplies. The
/// cluster cards contain an AuthorizeView (the Task 10 failover control), and
/// AuthorizeView throws without that cascading value.
/// </summary>
private IRenderedComponent<HealthPage> RenderHealthPage()
{
var host = Render<CascadingAuthenticationState>(parameters => parameters
.Add(p => p.ChildContent, (RenderFragment)(builder =>
{
builder.OpenComponent<HealthPage>(0);
builder.CloseComponent();
})));
return host.FindComponent<HealthPage>();
}
[Fact]
public void Renders_OutboxKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -129,7 +163,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheNotificationKpisPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/notifications/kpis']");
Assert.Contains("View details", link.TextContent);
}
@@ -148,7 +182,7 @@ public class HealthPageTests : BunitContext
AsOfUtc: DateTime.UtcNow)));
Services.AddSingleton(auditService);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -166,7 +200,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void Renders_SiteCallKpiTiles_WithValues()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
// KPI data arrives via an async actor Ask after first render.
cut.WaitForAssertion(() =>
@@ -186,7 +220,7 @@ public class HealthPageTests : BunitContext
[Fact]
public void RendersLinkToTheSiteCallsReportPage()
{
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
var link = cut.Find("a[href='/site-calls/report']");
Assert.Contains("View details", link.TextContent);
}
@@ -197,7 +231,7 @@ public class HealthPageTests : BunitContext
_siteCallKpiReply = new SiteCallKpiResponse(
"k", false, "site call repository unavailable", 0, 0, 0, 0, null, 0);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -216,7 +250,7 @@ public class HealthPageTests : BunitContext
_kpiReply = new NotificationKpiResponse(
"k", false, "outbox repository unavailable", 0, 0, 0, 0, null);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -234,7 +268,7 @@ public class HealthPageTests : BunitContext
// default-site load produces charts.
SeedSites("site-a");
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -268,7 +302,7 @@ public class HealthPageTests : BunitContext
throw new InvalidOperationException("kpi history unavailable"));
Services.AddSingleton(faulting);
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -294,7 +328,7 @@ public class HealthPageTests : BunitContext
LastReportReceivedAt = DateTimeOffset.UtcNow.AddMinutes(-5),
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
@@ -316,7 +350,7 @@ public class HealthPageTests : BunitContext
LastStatusChangeAt = changedAt,
});
var cut = Render<HealthPage>();
var cut = RenderHealthPage();
cut.WaitForAssertion(() =>
{
tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteCommunicationActorTests.cs
+85
View File
@@ -372,4 +372,89 @@ public class SiteCommunicationActorTests : TestKit
Assert.Equal(isActive, heartbeat.IsActive);
Assert.Equal("site1", heartbeat.SiteId);
}
// ---- Central→site failover relay (Task 10) ----------------------------------
// The failover action is injected for the same reason isActiveCheck is: performing a
// real Cluster.Leave would require Akka.Cluster in the TestKit ActorSystem. These
// tests pin the ROUTING and GUARD logic; the actual Leave against a real two-node
// site cluster is covered by SiteFailoverRelayTests in the integration suite.
[Fact]
public void TriggerSiteFailover_IssuesTheLeave_AndAcksWithTheTarget()
{
var dmProbe = CreateTestProbe();
string? roleAskedFor = null;
Func<string, string?> failOver = role =>
{
roleAskedFor = role;
return "akka.tcp://scadabridge@site1-a:8082";
};
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-1", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.True(ack.Accepted);
Assert.Equal("corr-1", ack.CorrelationId);
Assert.Equal("akka.tcp://scadabridge@site1-a:8082", ack.TargetAddress);
Assert.Null(ack.ErrorMessage);
// Site singletons are scoped to the site-specific role, not the base "Site" role;
// failing over the wrong scope would move the wrong node.
Assert.Equal("site-site1", roleAskedFor);
}
[Fact]
public void TriggerSiteFailover_RefusesWhenThereIsNoPeer()
{
var dmProbe = CreateTestProbe();
Func<string, string?> failOver = _ => null;
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-2", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Null(ack.TargetAddress);
Assert.NotNull(ack.ErrorMessage);
}
[Fact]
public void TriggerSiteFailover_RefusesACommandAddressedToAnotherSite()
{
// A misrouted command must be refused, not silently acted on — acting would fail
// over a site the operator never selected.
var dmProbe = CreateTestProbe();
var invoked = false;
Func<string, string?> failOver = _ =>
{
invoked = true;
return "addr";
};
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-3", "site2"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Contains("site2", ack.ErrorMessage);
Assert.False(invoked);
}
[Fact]
public void TriggerSiteFailover_FaultInTheLeave_IsReportedNotThrown()
{
var dmProbe = CreateTestProbe();
Func<string, string?> failOver = _ => throw new InvalidOperationException("cluster unavailable");
var siteActor = Sys.ActorOf(Props.Create(() =>
new SiteCommunicationActor("site1", _options, dmProbe.Ref, null, failOver)));
siteActor.Tell(new TriggerSiteFailover("corr-4", "site1"));
var ack = ExpectMsg<SiteFailoverAck>();
Assert.False(ack.Accepted);
Assert.Contains("cluster unavailable", ack.ErrorMessage);
}
}
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SiteFailoverRelayTests.cs
+51
View File
@@ -0,0 +1,51 @@
using Akka.Actor;
using Akka.Cluster;
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// Real-cluster proof for the central→site failover relay (Task 10). The unit tests in
/// <c>SiteCommunicationActorTests</c> pin the routing and guard logic with the failover action
/// stubbed out; this pins the part they cannot — that the shared
/// <see cref="ClusterFailoverCoordinator"/> actually moves a SITE pair when scoped to the
/// site-specific role.
///
/// <para>The site-specific role scope is the load-bearing detail: site singletons (the
/// Deployment Manager) are placed on <c>site-{SiteId}</c>, not on the base <c>Site</c> role, so
/// failing over the wrong scope would move the wrong node.</para>
/// </summary>
public sealed class SiteFailoverRelayTests
{
[Fact]
public async Task Failing_over_a_site_pair_moves_the_oldest_and_the_survivor_takes_over()
{
// A site pair, both nodes carrying the site-specific role.
await using var f = await TwoNodeClusterFixture.StartAsync(role: "site-SiteA");
var oldest = Akka.Cluster.Cluster.Get(f.NodeA);
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(oldest, "site-SiteA"),
"precondition: NodeA must be the active (oldest Up) site node");
// Issued from the other node, as the relay does when contact rotation lands there.
var target = ClusterFailoverCoordinator.FailOverOldest(f.NodeB, "site-SiteA");
Assert.Equal(oldest.SelfAddress, target);
await f.NodeA.WhenTerminated.WaitAsync(TimeSpan.FromSeconds(30));
await TwoNodeClusterFixture.WaitForMemberRemoved(f.NodeB, oldest.SelfAddress, TimeSpan.FromSeconds(30));
Assert.True(ActiveNodeEvaluator.SelfIsOldestUp(Akka.Cluster.Cluster.Get(f.NodeB), "site-SiteA"));
}
[Fact]
public async Task A_site_role_scope_that_matches_no_member_is_refused()
{
// Guards the role-scoping mistake directly: asking for a site that isn't this pair
// must find no members and refuse, rather than falling back to some other node.
await using var f = await TwoNodeClusterFixture.StartAsync(role: "site-SiteA");
var target = ClusterFailoverCoordinator.FailOverOldest(f.NodeB, "site-SiteB");
Assert.Null(target);
// Positive control: the pair is untouched and still fully formed.
await TwoNodeClusterFixture.WaitForMembersUp(f.NodeB, 2, TimeSpan.FromSeconds(10));
}
}