feat(cluster): manual central failover service — graceful Leave of the oldest Up member

Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
This commit is contained in:
Joseph Doherty
2026-07-22 06:55:15 -04:00
parent eca69505bc
commit f679d5c749
8 changed files with 316 additions and 8 deletions
@@ -3,7 +3,8 @@ namespace ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
/// <summary>
/// Top-level Audit Log channel — the trust boundary the audited action crosses.
/// One of: outbound API call, outbound DB write, notification send/deliver, inbound API request,
/// or a two-person ("secured") write through its submit/approve/reject/execute lifecycle.
/// a two-person ("secured") write through its submit/approve/reject/execute lifecycle, or an
/// operator-initiated cluster topology action.
/// </summary>
public enum AuditChannel
{
@@ -11,5 +12,14 @@ public enum AuditChannel
DbOutbound,
Notification,
ApiInbound,
SecuredWrite
SecuredWrite,
/// <summary>
/// An operator-initiated change to cluster topology — currently only the admin-triggered
/// manual failover of the central pair. Distinct from the script trust boundary the other
/// channels describe: nothing here crosses into user script, but a human deliberately
/// restarted the active node, which is exactly the kind of act an audit log exists to
/// attribute. (decision 2026-07-22)
/// </summary>
Cluster
}
@@ -40,5 +40,14 @@ public enum AuditKind
/// <c>EventId</c>, source site, and final error) so the loss is queryable in
/// the Audit Log itself, not only in a rotating log file.
/// </summary>
ReconciliationAbandoned
ReconciliationAbandoned,
/// <summary>
/// An administrator triggered a manual failover of the central pair from the Health page:
/// the active (oldest Up) node was asked to leave the cluster gracefully so its singletons
/// hand over and the standby takes over. One row per invocation, written BEFORE the Leave
/// is issued (the acting node is usually not the one that goes away, but the audit must
/// survive either outcome). (decision 2026-07-22)
/// </summary>
ManualFailover
}