dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(cluster): site-pair manual failover relayed from the central UI (Task 10)

Browse Source 
Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
This commit is contained in:
Joseph Doherty
2026-07-22 07:48:56 -04:00
parent d66e0d585f
commit c8e2f4da02
12 changed files with 621 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.ScadaBridge.Host/Health/AkkaManualFailoverService.cs
+56 -26
View File
@@ -6,6 +6,8 @@ using ZB.MOM.WW.ScadaBridge.CentralUI.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Services;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Audit;
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
using ZB.MOM.WW.ScadaBridge.Communication;
using ZB.MOM.WW.ScadaBridge.Communication.ClusterState;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.Host.Health;
@@ -33,19 +35,23 @@ public sealed class AkkaManualFailoverService : IManualFailoverService
{
private readonly AkkaHostedService _akka;
private readonly ICentralAuditWriter _audit;
private readonly CommunicationService _communication;
private readonly ILogger<AkkaManualFailoverService> _logger;
/// <summary>Initializes a new <see cref="AkkaManualFailoverService"/>.</summary>
/// <param name="akka">The Akka hosted service exposing the cluster's actor system.</param>
/// <param name="audit">Central direct-write audit writer.</param>
/// <param name="communication">Central→site command/control transport, used for site failover.</param>
/// <param name="logger">Logger.</param>
public AkkaManualFailoverService(
AkkaHostedService akka,
ICentralAuditWriter audit,
CommunicationService communication,
ILogger<AkkaManualFailoverService> logger)
{
_akka = akka;
_audit = audit;
_communication = communication;
_logger = logger;
}
@@ -76,46 +82,63 @@ public sealed class AkkaManualFailoverService : IManualFailoverService
return target.ToString();
}
/// <inheritdoc />
public async Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor)
{
// Audit BEFORE relaying, for the same reason as central: the row must exist even if
// the outcome is never observed. Unlike central, the acting node is never the one
// leaving — but a relay can still time out, and an un-acked request that DID take
// effect at the site would otherwise be unattributed.
await WriteAuditAsync(actor, target: siteId, sourceSiteId: siteId);
_logger.LogWarning(
"Manual site failover triggered by {Actor} for site {SiteId}; relaying to the site cluster.",
actor, siteId);
try
{
var ack = await _communication.TriggerSiteFailoverAsync(siteId, Guid.NewGuid().ToString());
return new SiteFailoverOutcome(ack.Accepted, ack.TargetAddress, ack.ErrorMessage);
}
catch (Exception ex)
{
// Central never buffers for an unreachable site — the Ask simply times out. Report
// that distinctly from a refusal, which is a definitive answer FROM the site.
_logger.LogWarning(ex,
"Manual site failover for {SiteId} did not get an ack from the site.", siteId);
return new SiteFailoverOutcome(
Accepted: false,
TargetAddress: null,
ErrorMessage: $"Site did not respond: {ex.Message}");
}
}
/// <summary>The Akka role scoping central-cluster membership.</summary>
private const string CentralRole = "Central";
/// <summary>
/// Oldest Up member with the role leaves — mirrors <c>ActiveNodeEvaluator</c>'s oldest-Up
/// rule so the node acted on is exactly the one hosting the singletons. Returns null when
/// fewer than 2 Up members carry the role (no peer = failover would be an outage).
/// Oldest Up member with the role leaves. Delegates to
/// <see cref="ClusterFailoverCoordinator.FailOverOldest"/>, which lives in Communication so
/// the site-pair failover path (inside <c>SiteCommunicationActor</c>, which cannot reference
/// Host) shares one implementation of the rule.
/// </summary>
/// <param name="system">The actor system whose cluster is acted on.</param>
/// <param name="role">Role scope for membership.</param>
/// <param name="dryRun">When true, resolve and return the target without issuing the Leave.</param>
/// <returns>The address that leaves (or would leave), or null when there is no peer.</returns>
public static Address? FailOverCore(ActorSystem system, string role, bool dryRun = false)
{
var cluster = Akka.Cluster.Cluster.Get(system);
var withRole = cluster.State.Members
.Where(m => m.Status == MemberStatus.Up && m.HasRole(role))
.OrderBy(m => m, Member.AgeOrdering)
.ToList();
if (withRole.Count < 2)
{
return null;
}
var oldest = withRole[0];
if (!dryRun)
{
cluster.Leave(oldest.Address);
}
return oldest.Address;
}
=> ClusterFailoverCoordinator.FailOverOldest(system, role, dryRun);
/// <summary>
/// Best-effort audit row. Audit failure must never block the failover the admin asked for
/// — the same rule the rest of the Audit Log follows (audit is best-effort; the
/// user-facing action's own success path is authoritative).
/// </summary>
private async Task WriteAuditAsync(string actor, Address target)
private Task WriteAuditAsync(string actor, Address target)
=> WriteAuditAsync(actor, target.ToString(), sourceSiteId: null);
/// <inheritdoc cref="WriteAuditAsync(string, Address)"/>
private async Task WriteAuditAsync(string actor, string target, string? sourceSiteId)
{
try
{
@@ -124,8 +147,15 @@ public sealed class AkkaManualFailoverService : IManualFailoverService
kind: AuditKind.ManualFailover,
status: AuditStatus.Submitted,
actor: actor,
target: target.ToString(),
extra: JsonSerializer.Serialize(new { target = target.ToString(), role = CentralRole }));
target: target,
sourceSiteId: sourceSiteId,
extra: JsonSerializer.Serialize(new
{
target,
// Central rows name the Central role; site rows name the site, so a query
// can tell which pair an operator moved.
scope = sourceSiteId is null ? CentralRole : $"site-{sourceSiteId}"
}));
await _audit.WriteAsync(evt);
}