dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(ui): admin manual-failover control on the health page

Browse Source 
CentralFailoverControl lives in Components/Health/ (matching AuditKpiTiles /
SiteCallKpiTiles) rather than inline in Health.razor, so it is testable without
standing up the whole dashboard's DI graph.

- Admin-gated via AuthorizeView + RequireAdmin. The Health page itself is
  intentionally all-roles, so the gate belongs on the control, not the page.
- Disabled with an explanatory title when the pair has no online standby;
  the authoritative guard remains server-side against live cluster membership.
- Confirmation dialog (IDialogService, the page idiom) warns that singletons
  hand over, in-flight work on the active node is interrupted, and THIS PAGE
  will disconnect and reconnect against the new active node -- Traefik routes
  the UI to the node being restarted, so a working failover otherwise reads as
  a crash the admin caused.
- A refused failover (service returns null) surfaces the refusal; the UI never
  reports a failover that did not happen.

7 bUnit tests. Two harness requirements that bit first: AuthorizeView needs a
cascading AuthenticationState (the app supplies it from the layout), and
BunitContext pre-registers a placeholder IAuthorizationService that throws on
policy evaluation -- both handled the same way NavMenuTests documents.

Runbook paragraphs added to Component-ClusterInfrastructure.md (new Manual
Failover section) and docker/README.md.
This commit is contained in:
Joseph Doherty
2026-07-22 07:00:10 -04:00
parent f679d5c749
commit caf14a3e03
5 changed files with 309 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/requirements/Component-ClusterInfrastructure.md
+15
View File
@@ -137,6 +137,21 @@ Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters bui
The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
### Manual Failover (admin-triggered)
An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).
Semantics:
- **Graceful `Leave`, never `Down`.** The active node leaves the cluster so `ClusterSingletonManager` hands its singletons to the standby before the member is removed. A `Down` would skip the hand-off.
- **Target = the oldest Up member with the `Central` role**, the same rule `ActiveNodeEvaluator` uses, so the node acted on is exactly the one hosting the singletons — never Akka's cluster *leader*, whose address-ordered definition diverges from singleton placement after a restart.
- **Peer guard.** Refused when fewer than two Up `Central` members exist: failing over a lone node is an outage, not a failover. Enforced server-side in `AkkaManualFailoverService`; the button is also disabled client-side when the pair has no online standby.
- **Admin-only** (`AuthorizationPolicies.RequireAdmin`). The Health page itself is all-roles, so the gate lives on the control.
- **Audited before acting.** One `AuditChannel.Cluster` / `AuditKind.ManualFailover` row is written through `ICentralAuditWriter` *before* the Leave is issued, naming the actor and the target address — the acting node can be the one that goes away, and an audit written afterwards could be lost to the shutdown it describes. Audit failure never blocks the failover.
- **Your own page will disconnect.** Traefik routes the UI to the active node, so triggering a failover drops the admin's Blazor circuit; it reconnects against the new active node. The confirmation dialog says so.
After the Leave the node's `ActorSystem` terminates, the `WhenTerminated` watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member (the new standby). Recovery is the normal restart contract above — no interaction with seed ordering, since the peer is alive on this path.
## Single-Node Operation
`akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.