feat(cluster): site-pair manual failover relayed from the central UI (Task 10)

Central and each site are SEPARATE Akka clusters, so central cannot act on a site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract travels the existing ClusterClient command/control channel (mirroring the RetryParkedOperation relay); the site's own SiteCommunicationActor performs the graceful Leave and acks the outcome. - ClusterFailoverCoordinator moved out of Host into Communication/ClusterState, beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation; SiteCommunicationActor cannot reference Host, and the two definitions must not drift or the node asked to leave stops being the singleton host. - Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role -- site singletons are placed on the former, so the base role would move the wrong node. Pinned by a unit test asserting the role string and by a real-cluster test. - Site-side guards: refuses a command addressed to another site (a misroute must never fail over a site the operator did not select), refuses when there is no peer, and reports a fault as an ack rather than throwing into supervision -- a restart there would drop central's Ask into a bare timeout and lose the reason. - Ack is sent before the Leave takes effect so it still reaches central. - UI: the same control now serves both scopes via a SiteId parameter. The site confirmation deliberately does NOT claim the admin's page will disconnect -- it won't, and crying wolf there devalues the central warning that is real. A site refusal and an unreachable site surface distinctly. - Rolling upgrade: a site on an older binary has no handler, so the message dead-letters and the Ask times out, reported as "site did not respond". That is the honest outcome; documented on the contract. Fallout fixed: HealthPageTests now renders the page inside CascadingAuthenticationState with the real policy set and IAuthorizationService, because the cards embed an AuthorizeView. That mirrors production, where the layout supplies the cascading value.
2026-07-22 07:48:56 -04:00
parent d66e0d585f
commit c8e2f4da02
12 changed files with 621 additions and 61 deletions
@@ -13,7 +13,9 @@
        <button class="btn btn-outline-warning btn-sm"
                disabled="@(!HasPeer || _busy)"
                title="@(HasPeer
-                          ? "Gracefully restart the active node; the standby takes over."
+                          ? (IsSite
+                              ? $"Gracefully restart site {SiteId}'s active node; its standby takes over."
+                              : "Gracefully restart the active node; the standby takes over.")
                          : "No standby available — failing over a lone node would be an outage, not a failover.")"
                @onclick="TriggerAsync">
            @(_busy ? "Failing over…" : "Trigger failover")
@@ -34,6 +36,17 @@
    [Parameter]
    public int OnlineCentralNodeCount { get; set; }

+    /// <summary>
+    /// When set, this control fails over the named SITE pair instead of the central pair.
+    /// Sites are separate Akka clusters reached over the command/control relay, so the site
+    /// path cannot disturb the admin's own session — which is why the confirmation text
+    /// differs. <c>null</c> (the default) means the central pair.
+    /// </summary>
+    [Parameter]
+    public string? SiteId { get; set; }
+
+    private bool IsSite => !string.IsNullOrWhiteSpace(SiteId);
+
    [CascadingParameter]
    private Task<AuthenticationState>? AuthState { get; set; }

@@ -45,16 +58,26 @@

    private async Task TriggerAsync()
    {
-        // The admin is almost certainly connected THROUGH the node about to restart (Traefik
-        // routes to the active node), so the dialog has to set that expectation or a working
-        // failover reads as a crash they just caused.
-        var confirmed = await Dialog.ConfirmAsync(
-            "Trigger central failover?",
-            "The active central node will leave the cluster and restart; the standby takes over "
-            + "and becomes active. Cluster singletons hand over gracefully, but in-flight work on "
-            + "the active node is interrupted. This page is served by the active node, so it will "
-            + "briefly disconnect and reconnect against the new active node.",
-            danger: true);
+        // Central: the admin is almost certainly connected THROUGH the node about to restart
+        // (Traefik routes to the active node), so the dialog must set that expectation or a
+        // working failover reads as a crash they caused.
+        // Site: a different cluster entirely — this page is unaffected, and claiming otherwise
+        // would train operators to ignore the warning that does matter.
+        var confirmed = IsSite
+            ? await Dialog.ConfirmAsync(
+                $"Trigger failover for site {SiteId}?",
+                $"The active node of site {SiteId} will leave its cluster and restart; the site's "
+                + "standby takes over and its Deployment Manager singleton hands over gracefully. "
+                + "In-flight work on that site node is interrupted, and the site is briefly "
+                + "unavailable while the handover completes.",
+                danger: true)
+            : await Dialog.ConfirmAsync(
+                "Trigger central failover?",
+                "The active central node will leave the cluster and restart; the standby takes over "
+                + "and becomes active. Cluster singletons hand over gracefully, but in-flight work on "
+                + "the active node is interrupted. This page is served by the active node, so it will "
+                + "briefly disconnect and reconnect against the new active node.",
+                danger: true);

        if (!confirmed)
        {
@@ -68,17 +91,35 @@
        try
        {
            var actor = await ResolveActorAsync();
-            var target = await Failover.FailOverCentralAsync(actor);

-            if (target is null)
+            if (IsSite)
            {
-                // The server-side peer guard refused. Never report a failover that did not happen.
-                _failed = true;
-                _message = "Refused: no standby available to take over.";
+                var outcome = await Failover.FailOverSiteAsync(SiteId!, actor);
+                if (outcome.Accepted)
+                {
+                    _message = $"Failover triggered — {outcome.TargetAddress} is leaving the site cluster.";
+                }
+                else
+                {
+                    // Carry the site's own words through: a peer-guard refusal and an
+                    // unreachable site are different situations for the operator.
+                    _failed = true;
+                    _message = outcome.ErrorMessage ?? "Refused by the site.";
+                }
            }
            else
            {
-                _message = $"Failover triggered — {target} is leaving the cluster.";
+                var target = await Failover.FailOverCentralAsync(actor);
+                if (target is null)
+                {
+                    // The server-side peer guard refused. Never report a failover that did not happen.
+                    _failed = true;
+                    _message = "Refused: no standby available to take over.";
+                }
+                else
+                {
+                    _message = $"Failover triggered — {target} is leaving the cluster.";
+                }
            }
        }
        catch (Exception ex)
@@ -219,12 +219,11 @@
                        }
                    </div>
                    <div class="d-flex align-items-center gap-3">
-                        @if (isCentral)
-                        {
-                            @* Admin-only; the control gates itself, and disables when the
-                               central pair has no online standby to take over. *@
-                            <CentralFailoverControl OnlineCentralNodeCount="@OnlineNodeCount(state)" />
-                        }
+                        @* Admin-only; the control gates itself, and disables when the pair has
+                           no online standby to take over. Central acts on the local cluster;
+                           a site is asked over the command/control relay. *@
+                        <CentralFailoverControl OnlineCentralNodeCount="@OnlineNodeCount(state)"
+                                                SiteId="@(isCentral ? null : siteId)" />
                        <small class="text-muted">
                            Last report: <TimestampDisplay Value="@state.LastReportReceivedAt" Format="HH:mm:ss" NullText="awaiting first report" />
                            | Last heartbeat: <TimestampDisplay Value="@state.LastHeartbeatAt" Format="HH:mm:ss" />
@@ -21,4 +21,28 @@ public interface IManualFailoverService
    /// <returns>The address acted on, or <c>null</c> when there is no peer to fail over to
    /// (failing over a lone node would be an outage, not a failover).</returns>
    Task<string?> FailOverCentralAsync(string actor);
+
+    /// <summary>
+    /// Asks a SITE to gracefully fail over its own two-node pair. Central and each site are
+    /// separate Akka clusters, so this is a request relayed over the command/control channel —
+    /// the site performs the Leave itself and reports the outcome. Unlike central failover,
+    /// this does NOT disturb the caller's own UI session.
+    /// </summary>
+    /// <param name="siteId">The site whose pair should fail over.</param>
+    /// <param name="actor">Authenticated user name, recorded on the audit row.</param>
+    /// <returns>The site's outcome — accepted with a target, or refused with a reason.</returns>
+    Task<SiteFailoverOutcome> FailOverSiteAsync(string siteId, string actor);
 }
+
+/// <summary>
+/// Outcome of a site-pair failover request, in UI terms. Deliberately distinct from the wire
+/// ack so CentralUI stays free of the Akka message contract.
+/// </summary>
+/// <param name="Accepted"><c>true</c> when the site issued the graceful Leave.</param>
+/// <param name="TargetAddress">Address of the leaving site node; <c>null</c> when refused.</param>
+/// <param name="ErrorMessage">
+/// Why the request was refused or failed. A refusal (peer guard, misroute) is a definitive
+/// answer FROM the site; an unreachable site surfaces here as a timeout message. Both are
+/// reported to the operator rather than being flattened into a generic failure.
+/// </param>
+public sealed record SiteFailoverOutcome(bool Accepted, string? TargetAddress, string? ErrorMessage);