feat(cluster): site-pair manual failover relayed from the central UI (Task 10)

Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.

- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
  beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
  SiteCommunicationActor cannot reference Host, and the two definitions must not
  drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
  site singletons are placed on the former, so the base role would move the wrong
  node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
  never fail over a site the operator did not select), refuses when there is no
  peer, and reports a fault as an ack rather than throwing into supervision --
  a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
  confirmation deliberately does NOT claim the admin's page will disconnect --
  it won't, and crying wolf there devalues the central warning that is real. A
  site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
  dead-letters and the Ask times out, reported as "site did not respond". That
  is the honest outcome; documented on the contract.

Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
This commit is contained in:
Joseph Doherty
2026-07-22 07:48:56 -04:00
parent d66e0d585f
commit c8e2f4da02
12 changed files with 621 additions and 61 deletions
@@ -0,0 +1,53 @@
namespace ZB.MOM.WW.ScadaBridge.Commons.Messages.RemoteQuery;
/// <summary>
/// Central → site relay command: gracefully fail over the owning site's two-node cluster.
/// Sent over the command/control channel when an Administrator clicks "Trigger failover" on a
/// site card in the Central UI Health dashboard.
/// </summary>
/// <remarks>
/// <para>
/// Central and each site are SEPARATE Akka clusters, so central cannot act on a site's
/// membership directly — it can only ask. The site's own <c>SiteCommunicationActor</c> performs
/// the <c>Cluster.Leave</c> against its site-specific <c>site-{SiteId}</c> role, which is the
/// role its singletons (the Deployment Manager) are scoped to.
/// </para>
/// <para>
/// Either site node may receive this: <c>SiteCommunicationActor</c> is registered per node
/// (not as a singleton), and ClusterClient contact rotation reaches whichever answers. That is
/// fine — <c>Cluster.Leave(address)</c> is valid from any member, and the target is resolved
/// from cluster state rather than from who received the message.
/// </para>
/// <para>
/// <b>Rolling upgrade:</b> a site running a binary older than this contract has no handler for
/// this message; it becomes an unhandled message / dead letter and central's Ask times out.
/// The timeout is reported to the operator as "site did not respond", which is the correct
/// user-facing outcome — an old site genuinely cannot honour the request. Message evolution
/// stays additive-only.
/// </para>
/// </remarks>
/// <param name="CorrelationId">Correlation id echoed on the ack.</param>
/// <param name="SiteId">The site whose pair should fail over. Carried explicitly so a
/// misrouted command is detectable at the site rather than silently acted on.</param>
public sealed record TriggerSiteFailover(
string CorrelationId,
string SiteId);
/// <summary>
/// Site → central ack for a <see cref="TriggerSiteFailover"/> relay command.
/// </summary>
/// <param name="CorrelationId">Correlation id of the originating relay command.</param>
/// <param name="Accepted">
/// <c>true</c> when the site resolved a target and issued the graceful Leave.
/// <c>false</c> is a definitive refusal from the site — most often the peer guard (fewer than
/// two Up members in the site role, so a failover would be an outage) or a site-id mismatch.
/// A <c>false</c> ack is NOT a transport failure and must be distinguished from an
/// unreachable-site timeout.
/// </param>
/// <param name="TargetAddress">Address of the node that is leaving; <c>null</c> when refused.</param>
/// <param name="ErrorMessage">Reason for a refusal, or a fault message; <c>null</c> on success.</param>
public sealed record SiteFailoverAck(
string CorrelationId,
bool Accepted,
string? TargetAddress,
string? ErrorMessage);