feat(cluster): manual central failover service — graceful Leave of the oldest Up member

Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.

- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
  the node acted on is exactly the one hosting the singletons (never the leader,
  whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
  node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
  the one that goes away, and an audit written after could be lost to the shutdown it
  describes. Best-effort — audit failure never blocks the failover.

New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.

alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.

ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
This commit is contained in:
Joseph Doherty
2026-07-22 06:55:15 -04:00
parent eca69505bc
commit f679d5c749
8 changed files with 316 additions and 8 deletions
@@ -12,7 +12,7 @@ public class AuditEnumTests
[Fact]
public void AuditChannel_HasExactlyExpectedMembers()
{
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite" };
var expected = new[] { "ApiOutbound", "DbOutbound", "Notification", "ApiInbound", "SecuredWrite", "Cluster" };
var actual = Enum.GetValues(typeof(AuditChannel))
.Cast<AuditChannel>()
.Select(x => x.ToString())
@@ -23,7 +23,7 @@ public class AuditEnumTests
}
[Fact]
public void AuditKind_HasExactlySixteenExpectedMembers()
public void AuditKind_HasExactlySeventeenExpectedMembers()
{
var expected = new[]
{
@@ -33,13 +33,14 @@ public class AuditEnumTests
"SecuredWriteSubmit", "SecuredWriteApprove", "SecuredWriteReject", "SecuredWriteExecute",
"SecuredWriteExpire",
"ReconciliationAbandoned",
"ManualFailover",
};
var actual = Enum.GetValues(typeof(AuditKind))
.Cast<AuditKind>()
.Select(x => x.ToString())
.ToArray();
Assert.Equal(16, actual.Length);
Assert.Equal(17, actual.Length);
Assert.Equal(expected, actual);
}