feat(cluster): manual central failover service — graceful Leave of the oldest Up member

Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
This commit is contained in:
Joseph Doherty
2026-07-22 06:55:15 -04:00
parent eca69505bc
commit f679d5c749
8 changed files with 316 additions and 8 deletions
@@ -0,0 +1,24 @@
namespace ZB.MOM.WW.ScadaBridge.CentralUI.Services;
/// <summary>
/// Admin-triggered manual failover of the central pair. Declared here — and in terms of
/// plain strings — so CentralUI stays Akka-free; the Akka implementation lives in the Host
/// (<c>AkkaManualFailoverService</c>) and is registered only in the Central branch.
/// </summary>
public interface IManualFailoverService
{
/// <summary>
/// Gracefully fails over the central cluster: the current active (oldest Up) member
/// leaves, restarts via its supervisor, and rejoins as standby. The Leave is graceful,
/// never a Down, so cluster singletons hand over instead of being killed.
/// <para>
/// The caller is usually connected THROUGH the node being failed over (Traefik routes
/// the UI to the active node), so the calling Blazor circuit should expect to drop and
/// reconnect against the new active node.
/// </para>
/// </summary>
/// <param name="actor">Authenticated user name, recorded on the audit row.</param>
/// <returns>The address acted on, or <c>null</c> when there is no peer to fail over to
/// (failing over a lone node would be an outage, not a failover).</returns>
Task<string?> FailOverCentralAsync(string actor);
}