Central and each site are SEPARATE Akka clusters, so central cannot act on a
site's membership -- it asks. New TriggerSiteFailover/SiteFailoverAck contract
travels the existing ClusterClient command/control channel (mirroring the
RetryParkedOperation relay); the site's own SiteCommunicationActor performs the
graceful Leave and acks the outcome.
- ClusterFailoverCoordinator moved out of Host into Communication/ClusterState,
beside ActiveNodeEvaluator. Both paths now share ONE oldest-Up implementation;
SiteCommunicationActor cannot reference Host, and the two definitions must not
drift or the node asked to leave stops being the singleton host.
- Site scope is the SITE-SPECIFIC role (site-{SiteId}), not the base Site role --
site singletons are placed on the former, so the base role would move the wrong
node. Pinned by a unit test asserting the role string and by a real-cluster test.
- Site-side guards: refuses a command addressed to another site (a misroute must
never fail over a site the operator did not select), refuses when there is no
peer, and reports a fault as an ack rather than throwing into supervision --
a restart there would drop central's Ask into a bare timeout and lose the reason.
- Ack is sent before the Leave takes effect so it still reaches central.
- UI: the same control now serves both scopes via a SiteId parameter. The site
confirmation deliberately does NOT claim the admin's page will disconnect --
it won't, and crying wolf there devalues the central warning that is real. A
site refusal and an unreachable site surface distinctly.
- Rolling upgrade: a site on an older binary has no handler, so the message
dead-letters and the Ask times out, reported as "site did not respond". That
is the honest outcome; documented on the contract.
Fallout fixed: HealthPageTests now renders the page inside
CascadingAuthenticationState with the real policy set and IAuthorizationService,
because the cards embed an AuthorizeView. That mirrors production, where the
layout supplies the cascading value.
Admin-triggered failover of the central pair. IManualFailoverService is declared in
CentralUI (plain strings, so that project stays Akka-free); AkkaManualFailoverService
implements it in the Host and is registered only in the Central branch.
- Leave, never Down: singletons hand over instead of being killed.
- Target = oldest Up member with the Central role, mirroring ActiveNodeEvaluator, so
the node acted on is exactly the one hosting the singletons (never the leader,
whose address-ordered definition diverges from singleton placement after a restart).
- Peer guard: returns null when fewer than 2 Up Central members — failing over a lone
node is an outage, not a failover.
- Audited BEFORE the Leave is issued via ICentralAuditWriter: the acting node can be
the one that goes away, and an audit written after could be lost to the shutdown it
describes. Best-effort — audit failure never blocks the failover.
New audit taxonomy: AuditChannel.Cluster + AuditKind.ManualFailover (operator-initiated
topology actions are not script trust-boundary crossings, but are exactly what an audit
log exists to attribute). Lock-in tests updated 5->6 channels, 16->17 kinds.
alog.md §4 updated per the lock-in tests' contract. Both tables were already stale --
the Channel row omitted SecuredWrite and the Kind table claimed 10 while the code had
16 -- so they are completed here, not merely appended to.
ManualFailoverTests: 3 real-cluster tests (oldest leaves + survivor takes over, peer
guard refuses with a positive still-running assert, dry-run probe does not perturb).
Joseph Doherty