docs(cluster): site-pair manual failover runbook + component spec
This commit is contained in:
@@ -351,3 +351,10 @@ docker start scadabridge-site-a-a
|
||||
Same pattern applies for site-b (`scadabridge-site-b-a`/`scadabridge-site-b-b`) and site-c (`scadabridge-site-c-a`/`scadabridge-site-c-b`).
|
||||
|
||||
Failover takes approximately 25 seconds (2s heartbeat + 10s detection threshold + 15s stable-after for split-brain resolver).
|
||||
|
||||
**Manual site failover from the UI (admin-only).** Each site card on `/monitoring/health` carries the same **Trigger failover** button as the central card. Central and each site are separate Akka clusters, so this is a *request* relayed over the ClusterClient command/control channel — the site's own communication actor performs the graceful `Leave` against its `site-{SiteId}` role and acks the result.
|
||||
|
||||
- Unlike central failover, this does **not** disconnect your page — a site is a different cluster.
|
||||
- A refusal from the site (no standby, or a command addressed to a different site) reads differently from an unreachable site (Ask timeout); the UI shows the site's own reason. Only the timeout leaves any doubt about whether the failover took effect.
|
||||
- A site running an older binary has no handler for the command, so it dead-letters and you see "site did not respond".
|
||||
- Each invocation writes a `Cluster` / `ManualFailover` audit row stamped with the site id.
|
||||
|
||||
@@ -40,16 +40,16 @@
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"subject": "Task 10 (OPTIONAL, ask user): site-pair failover via central-to-site transport",
|
||||
"status": "pending",
|
||||
"subject": "Task 10: site-pair failover via central\u2192site transport (BUILT \u2014 user approved 2026-07-22)",
|
||||
"status": "completed",
|
||||
"blockedBy": [
|
||||
4
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"subject": "FOLLOW-UP (owner deferred): correct the false InitJoin safety claim in scadaproj + OtOpcUa design docs",
|
||||
"status": "pending"
|
||||
"subject": "FOLLOW-UP: OtOpcUa converged on self-first seed ordering (2 local commits, unpushed); scadaproj CLAUDE.md index rows still owner-held",
|
||||
"status": "completed"
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
|
||||
@@ -152,6 +152,24 @@ Semantics:
|
||||
|
||||
After the Leave the node's `ActorSystem` terminates, the `WhenTerminated` watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member (the new standby). Recovery is the normal restart contract above — no interaction with seed ordering, since the peer is alive on this path.
|
||||
|
||||
#### Site-pair failover
|
||||
|
||||
The same control appears on each **site** card. Central and each site are separate Akka clusters, so central cannot act on a site's membership — it *asks*, over the existing ClusterClient command/control channel:
|
||||
|
||||
1. `CommunicationService.TriggerSiteFailoverAsync` sends a `TriggerSiteFailover` inside a `SiteEnvelope`.
|
||||
2. The site's `SiteCommunicationActor` (registered per node, so contact rotation reaches whichever answers) resolves the target from cluster state and issues the graceful `Leave` locally.
|
||||
3. It replies `SiteFailoverAck` — sent **before** the Leave takes effect, so the ack still arrives when the acking node is the one leaving.
|
||||
|
||||
Differences from central failover, all deliberate:
|
||||
|
||||
- **Role scope is `site-{SiteId}`, not the base `Site` role.** Site singletons (the Deployment Manager) are placed on the site-specific role; using the base role would move the wrong node. Pinned by both a unit test asserting the role string and a real-cluster test.
|
||||
- **Misroute is refused.** A command whose `SiteId` does not match the receiving node's is rejected rather than acted on — a misroute must never fail over a site the operator did not select.
|
||||
- **A fault is acked, not thrown.** Reporting through the ack keeps the reason; letting it reach supervision would restart the communication actor and reduce central's Ask to a bare timeout.
|
||||
- **The operator's own session is unaffected** — a different cluster entirely. The confirmation dialog therefore does *not* carry the "this page will disconnect" warning that the central one does.
|
||||
- **Refusal vs unreachable are distinct.** A `false` ack is a definitive answer from the site (peer guard, misroute); a timeout means the site never answered. Both surface to the operator with their own wording, because only one of them means "nothing happened".
|
||||
|
||||
**Rolling upgrade.** A site running a binary older than this contract has no handler for `TriggerSiteFailover`; the message dead-letters and central's Ask times out, reported as "site did not respond". That is the correct user-facing outcome — an old site genuinely cannot honour the request. Message evolution stays additive-only.
|
||||
|
||||
## Single-Node Operation
|
||||
|
||||
`akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.
|
||||
|
||||
Reference in New Issue
Block a user