# ScadaBridge Failover Procedures

## Automatic Failover (No Intervention Required)

### Central Cluster Failover

**What happens automatically:**

1. Active central node becomes unreachable (process crash, network failure, hardware failure).
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
4. Standby node is promoted to active. Total time: ~25 seconds.
5. Cluster singletons migrate to the new active node.
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
7. Active user sessions continue (the cookie's data-protected ticket is validated by the new node via the shared Data Protection key ring; CLI bearer tokens via the shared signing key).
8. SignalR connections are dropped and Blazor clients automatically reconnect.

**What is preserved:**
- All configuration and deployment state (stored in SQL Server)
- Active sessions (shared Data Protection keys for cookies, shared signing key for CLI bearer tokens)
- Deployment status records (SQL Server with optimistic concurrency)

**What is temporarily disrupted:**
- In-flight deployments: Central re-queries site state and re-issues if needed (idempotent)
- Real-time debug view streams: Clients reconnect automatically
- Health dashboard: Resumes on reconnect

### Site Cluster Failover

**What happens automatically:**

1. Active site node becomes unreachable.
2. Failure detection and split-brain resolution (~25 seconds total).
3. Site Deployment Manager singleton migrates to standby.
4. Instance Actors are recreated from persisted SQLite configurations.
5. Staggered startup: 50ms delay between instance creations to prevent reconnection storms.
6. DCL connection actors reconnect to OPC UA servers.
7. Script Actors and Alarm Actors resume processing from incoming values (no stale state).
8. S&F buffer is read from SQLite — pending retries resume.

**What is preserved:**
- Deployed instance configurations (SQLite)
- Static attribute overrides (SQLite)
- S&F message buffer (SQLite)
- Site event logs (SQLite)

**What is temporarily disrupted:**
- Tag value subscriptions: DCL reconnects and re-subscribes transparently
- Active script executions: Cancelled; trigger fires again on next value change
- Alarm states: Re-evaluated from incoming tag values (correct state within one update cycle)

## Manual Intervention Scenarios

### Scenario 1: Both Central Nodes Down

**Symptoms:** No central UI access, sites report "central unreachable" in logs.

**Recovery:**
1. Start either central node. It will form a single-node cluster.
2. Verify SQL Server is accessible.
3. Check `/health/ready` returns 200.
4. Start the second node. It will join the cluster automatically.
5. Verify both nodes appear in the Akka.NET cluster member list (check logs for "Member joined").

**No data loss:** All state is in SQL Server.

### Scenario 2: Both Site Nodes Down

**Symptoms:** Site appears offline in central health dashboard.

**Recovery:**
1. Start either site node.
2. Check logs for "Store-and-forward SQLite storage initialized".
3. Verify instance actors are recreated: "Instance {Name}: created N script actors and M alarm actors".
4. Start the second site node.
5. Verify the site appears online in the central health dashboard within 60 seconds.

**No data loss:** All state is in SQLite.

### Scenario 3: Split-Brain (Network Partition Between Peers)

**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".

**How the system handles it:**
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
- Stable-after (15s): the window of sustained unreachability before the downing decision.

**Manual intervention (if auto-resolution fails):**
1. Stop both nodes.
2. Start the preferred node first (it becomes the "oldest").
3. Start the second node.

### Scenario 4: SQL Server Outage (Central)

**Symptoms:** Central UI returns errors. `/health/ready` returns 503. Logs show database connection failures.

**Impact:**
- Active sessions (valid cookie or bearer token) can still access cached UI state.
- New logins fail (LDAP auth still works but role mapping requires DB).
- Template changes and deployments fail.
- Sites continue operating independently.

**Recovery:**
1. Restore SQL Server access.
2. Central nodes will automatically reconnect (EF Core connection resiliency).
3. Verify `/health/ready` returns 200.
4. No manual intervention needed on ScadaBridge nodes.

### Scenario 5: Forced Singleton Migration

**When to use:** The active node is degraded but not crashed (e.g., high CPU, disk full).

**Procedure:**
1. Initiate graceful shutdown on the degraded node:
   - Stop the Windows Service: `sc.exe stop ScadaBridge-Central`
   - CoordinatedShutdown will migrate singletons to the standby.
2. Wait for the standby to take over (check logs for "Singleton acquired").
3. Fix the issue on the original node.
4. Restart the service. It will rejoin as standby.

## Failover Timeline

```
T+0s    Node failure detected (heartbeat timeout)
T+2s    Akka.NET marks node as unreachable
T+10s   Failure detection confirmed (threshold reached)
T+10s   Split-brain resolver begins stable-after countdown
T+25s   Resolver actions: surviving node promoted
T+25s   Singleton migration begins
T+26s   Instance Actors start recreating (staggered)
T+30s   Health report sent from new active node
T+60s   All instances operational (500 instances * 50ms stagger = 25s)
```