docs(cluster): site-pair manual failover runbook + component spec

This commit is contained in:
Joseph Doherty
2026-07-22 07:49:51 -04:00
parent c8e2f4da02
commit 654df8abc2
3 changed files with 29 additions and 4 deletions
+7
View File
@@ -351,3 +351,10 @@ docker start scadabridge-site-a-a
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.