Joseph Doherty
135 lines
5.8 KiB
Markdown
135 lines
5.8 KiB
Markdown
# ScadaBridge Failover Procedures
|
|
|
|
## Automatic Failover (No Intervention Required)
|
|
|
|
### Central Cluster Failover
|
|
|
|
**What happens automatically:**
|
|
|
|
1. Active central node becomes unreachable (process crash, network failure, hardware failure).
|
|
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
|
|
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
|
|
4. Standby node is promoted to active. Total time: ~25 seconds.
|
|
5. Cluster singletons migrate to the new active node.
|
|
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
|
|
7. Active user sessions continue (the cookie's data-protected ticket is validated by the new node via the shared Data Protection key ring; CLI bearer tokens via the shared signing key).
|
|
8. SignalR connections are dropped and Blazor clients automatically reconnect.
|
|
|
|
**What is preserved:**
|
|
- All configuration and deployment state (stored in SQL Server)
|
|
- Active sessions (shared Data Protection keys for cookies, shared signing key for CLI bearer tokens)
|
|
- Deployment status records (SQL Server with optimistic concurrency)
|
|
|
|
**What is temporarily disrupted:**
|
|
- In-flight deployments: Central re-queries site state and re-issues if needed (idempotent)
|
|
- Real-time debug view streams: Clients reconnect automatically
|
|
- Health dashboard: Resumes on reconnect
|
|
|
|
### Site Cluster Failover
|
|
|
|
**What happens automatically:**
|
|
|
|
1. Active site node becomes unreachable.
|
|
2. Failure detection and split-brain resolution (~25 seconds total).
|
|
3. Site Deployment Manager singleton migrates to standby.
|
|
4. Instance Actors are recreated from persisted SQLite configurations.
|
|
5. Staggered startup: 50ms delay between instance creations to prevent reconnection storms.
|
|
6. DCL connection actors reconnect to OPC UA servers.
|
|
7. Script Actors and Alarm Actors resume processing from incoming values (no stale state).
|
|
8. S&F buffer is read from SQLite — pending retries resume.
|
|
|
|
**What is preserved:**
|
|
- Deployed instance configurations (SQLite)
|
|
- Static attribute overrides (SQLite)
|
|
- S&F message buffer (SQLite)
|
|
- Site event logs (SQLite)
|
|
|
|
**What is temporarily disrupted:**
|
|
- Tag value subscriptions: DCL reconnects and re-subscribes transparently
|
|
- Active script executions: Cancelled; trigger fires again on next value change
|
|
- Alarm states: Re-evaluated from incoming tag values (correct state within one update cycle)
|
|
|
|
## Manual Intervention Scenarios
|
|
|
|
### Scenario 1: Both Central Nodes Down
|
|
|
|
**Symptoms:** No central UI access, sites report "central unreachable" in logs.
|
|
|
|
**Recovery:**
|
|
1. Start either central node. It will form a single-node cluster.
|
|
2. Verify SQL Server is accessible.
|
|
3. Check `/health/ready` returns 200.
|
|
4. Start the second node. It will join the cluster automatically.
|
|
5. Verify both nodes appear in the Akka.NET cluster member list (check logs for "Member joined").
|
|
|
|
**No data loss:** All state is in SQL Server.
|
|
|
|
### Scenario 2: Both Site Nodes Down
|
|
|
|
**Symptoms:** Site appears offline in central health dashboard.
|
|
|
|
**Recovery:**
|
|
1. Start either site node.
|
|
2. Check logs for "Store-and-forward SQLite storage initialized".
|
|
3. Verify instance actors are recreated: "Instance {Name}: created N script actors and M alarm actors".
|
|
4. Start the second site node.
|
|
5. Verify the site appears online in the central health dashboard within 60 seconds.
|
|
|
|
**No data loss:** All state is in SQLite.
|
|
|
|
### Scenario 3: Split-Brain (Network Partition Between Peers)
|
|
|
|
**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".
|
|
|
|
**How the system handles it:**
|
|
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
|
|
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
|
|
- Stable-after (15s): the window of sustained unreachability before the downing decision.
|
|
|
|
**Manual intervention (if auto-resolution fails):**
|
|
1. Stop both nodes.
|
|
2. Start the preferred node first (it becomes the "oldest").
|
|
3. Start the second node.
|
|
|
|
### Scenario 4: SQL Server Outage (Central)
|
|
|
|
**Symptoms:** Central UI returns errors. `/health/ready` returns 503. Logs show database connection failures.
|
|
|
|
**Impact:**
|
|
- Active sessions (valid cookie or bearer token) can still access cached UI state.
|
|
- New logins fail (LDAP auth still works but role mapping requires DB).
|
|
- Template changes and deployments fail.
|
|
- Sites continue operating independently.
|
|
|
|
**Recovery:**
|
|
1. Restore SQL Server access.
|
|
2. Central nodes will automatically reconnect (EF Core connection resiliency).
|
|
3. Verify `/health/ready` returns 200.
|
|
4. No manual intervention needed on ScadaBridge nodes.
|
|
|
|
### Scenario 5: Forced Singleton Migration
|
|
|
|
**When to use:** The active node is degraded but not crashed (e.g., high CPU, disk full).
|
|
|
|
**Procedure:**
|
|
1. Initiate graceful shutdown on the degraded node:
|
|
- Stop the Windows Service: `sc.exe stop ScadaBridge-Central`
|
|
- CoordinatedShutdown will migrate singletons to the standby.
|
|
2. Wait for the standby to take over (check logs for "Singleton acquired").
|
|
3. Fix the issue on the original node.
|
|
4. Restart the service. It will rejoin as standby.
|
|
|
|
## Failover Timeline
|
|
|
|
```
|
|
T+0s Node failure detected (heartbeat timeout)
|
|
T+2s Akka.NET marks node as unreachable
|
|
T+10s Failure detection confirmed (threshold reached)
|
|
T+10s Split-brain resolver begins stable-after countdown
|
|
T+25s Resolver actions: surviving node promoted
|
|
T+25s Singleton migration begins
|
|
T+26s Instance Actors start recreating (staggered)
|
|
T+30s Health report sent from new active node
|
|
T+60s All instances operational (500 instances * 50ms stagger = 25s)
|
|
```
|