Refine Cluster Infrastructure: split-brain, seed nodes, failure detection, dual recovery

Add keep-oldest split-brain resolver with 15s stable-after duration. Configure both
nodes as seed nodes for symmetric startup. Set moderate failure detection defaults
(2s heartbeat, 10s threshold, ~25s total failover). Document automatic dual-node
recovery from persistent storage with no manual intervention.

Component-ClusterInfrastructure.md

@@ -55,10 +55,41 @@ Both central and site clusters.
 - Health reporting resumes from the new active node.
 - Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
 
+## Split-Brain Resolution
+
+The system uses the Akka.NET **keep-oldest** split-brain resolver strategy:
+
+- On a network partition, the node that has been in the cluster longest remains active. The younger node downs itself.
+- **Stable-after duration**: 15 seconds. The cluster membership must remain stable (no changes) for 15 seconds before the resolver acts to down unreachable nodes. This prevents premature downing during startup or rolling restarts.
+- **Why keep-oldest**: With only two nodes, quorum-based strategies (static-quorum, keep-majority) cannot distinguish "one node crashed" from "network partition" — both sides see fewer than quorum and both would down themselves, resulting in total cluster shutdown. Keep-oldest accepts a brief potential dual-active window during true network partitions, which is safe because site state rebuilds from SQLite and central state is in MS SQL.
+
+## Failure Detection Timing
+
+Configurable defaults for heartbeat and failure detection:
+
+| Setting | Default | Description |
+|---------|---------|-------------|
+| Heartbeat interval | 2 seconds | Frequency of health check messages between nodes |
+| Failure detection threshold | 10 seconds | Time without heartbeat before a node is considered unreachable |
+| Stable-after (split-brain) | 15 seconds | Time cluster must be stable before resolver acts |
+| **Total failover time** | **~25 seconds** | Detection (10s) + stable-after (15s) + singleton restart |
+
+These values balance failover speed with stability — fast enough that data collection gaps are small, tolerant enough that brief network hiccups don't trigger unnecessary failovers.
+
+## Dual-Node Recovery
+
+If both nodes in a cluster fail simultaneously (e.g., site power outage):
+
+1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
+2. **State recovery**:
+   - **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
+   - **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
+3. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.
+
 ## Node Configuration
 
 Each node is configured with:
-- **Cluster seed nodes**: Addresses of both nodes in the cluster.
+- **Cluster seed nodes**: **Both nodes** are seed nodes — each node lists both itself and its partner. Either node can start first and form the cluster; the other joins when it starts. No startup ordering dependency.
 - **Cluster role**: Central or Site (plus site identifier for site clusters).
 - **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication.
 - **Local storage paths**: SQLite database locations (site nodes only).

Component-Communication.md

@@ -82,6 +82,40 @@ Central Cluster
 - Sites do **not** communicate with each other.
 - All inter-cluster communication flows through central.
 
+## Message Timeouts
+
+Each request/response pattern has a default timeout that can be overridden in configuration:
+
+| Pattern | Default Timeout | Rationale |
+|---------|----------------|-----------|
+| 1. Deployment | 120 seconds | Script compilation at the site can be slow |
+| 2. Instance Lifecycle | 30 seconds | Stop/start actors is fast |
+| 3. System-Wide Artifacts | 120 seconds per site | Includes shared script recompilation |
+| 4. Integration Routing | 30 seconds | External system waiting for response; Inbound API per-method timeout may cap this further |
+| 5. Recipe/Command Delivery | 30 seconds | Fire-and-forget with ack |
+| 8. Remote Queries | 30 seconds | Querying parked messages or event logs |
+
+Timeouts use the Akka.NET **ask pattern**. If no response is received within the timeout, the caller receives a timeout failure.
+
+## Transport Configuration
+
+Akka.NET remoting provides built-in connection management and failure detection. The following transport-level settings are **explicitly configured** (not left to framework defaults) for predictable behavior:
+
+- **Transport heartbeat interval**: Configurable interval at which heartbeat messages are sent over remoting connections (e.g., every 5 seconds).
+- **Failure detection threshold**: Number of missed heartbeats before the connection is considered lost (e.g., 3 missed heartbeats = 15 seconds with a 5-second interval).
+- **Reconnection**: Akka.NET remoting handles reconnection automatically. No custom reconnection logic is required.
+
+These settings should be tuned for the expected network conditions between central and site clusters.
+
+## Message Ordering
+
+Akka.NET guarantees message ordering between a specific sender/receiver actor pair. The Communication Layer relies on this guarantee — messages to a given site are processed in the order they are sent. Callers do not need to handle out-of-order delivery.
+
+## Connection Failure Behavior
+
+- **In-flight messages**: When a connection drops while a request is in flight (e.g., deployment sent but no response received), the Akka ask pattern times out and the caller receives a failure. There is **no automatic retry or buffering at central** — the engineer sees the failure in the UI and re-initiates the action. This is consistent with the design principle that central does not buffer messages.
+- **Debug streams**: Any connection interruption (failover or network blip) kills the debug stream. The engineer must reopen the debug view in the Central UI to re-establish the subscription with a fresh snapshot. There is no auto-resume.
+
 ## Failover Behavior
 
 - **Central failover**: The standby node takes over the Akka.NET cluster role. In-progress deployments are treated as failed. Sites reconnect to the new active central node.

docs/plans/2026-03-16-cluster-infrastructure-refinement-design.md

@@ -0,0 +1,45 @@
+# Cluster Infrastructure Refinement — Design
+
+**Date**: 2026-03-16
+**Component**: Cluster Infrastructure (`Component-ClusterInfrastructure.md`)
+**Status**: Approved
+
+## Problem
+
+The Cluster Infrastructure doc covered topology and failover behavior but lacked specification for the split-brain resolver strategy, seed node configuration, failure detection timing, and dual-node failure recovery.
+
+## Decisions
+
+### Split-Brain Resolver
+- **Keep-oldest** strategy. The longest-running node stays active on partition; the younger node downs itself.
+- Stable-after duration: 15 seconds — prevents premature downing during startup or transient instability.
+- Quorum-based strategies rejected because they cause total cluster shutdown on any partition in a two-node cluster.
+
+### Seed Node Configuration
+- **Both nodes are seed nodes.** No startup ordering dependency. Whichever node starts first forms the cluster.
+
+### Failure Detection Timing
+- Heartbeat interval: **2 seconds**.
+- Failure threshold: **10 seconds** (5 missed heartbeats).
+- Total failover time: **~25 seconds** (10s detection + 15s stable-after + singleton restart).
+- All values configurable. Defaults balance failover speed with stability.
+
+### Dual-Node Recovery
+- **Automatic recovery**, no manual intervention. First node up forms a new cluster from seed configuration.
+- Site clusters rebuild from SQLite (deployed configs, S&F buffer). Alarm states re-evaluate from live data.
+- Central cluster rebuilds from MS SQL. No message buffer state to recover.
+
+## Affected Documents
+
+| Document | Change |
+|----------|--------|
+| `Component-ClusterInfrastructure.md` | Added 3 new sections: Split-Brain Resolution, Failure Detection Timing, Dual-Node Recovery. Updated Node Configuration to clarify both-as-seed. |
+
+## Alternatives Considered
+
+- **Static-quorum / keep-majority**: Rejected — both cause total cluster shutdown on partition in a two-node cluster. Unacceptable for SCADA availability.
+- **Single designated seed node**: Rejected — creates startup ordering dependency for no benefit in a two-node cluster.
+- **Manual recovery on dual failure**: Rejected — system already persists all state needed for automatic recovery.
+- **Fast detection (1s/5s)**: Rejected — too sensitive; brief network hiccups would trigger unnecessary failovers and full actor hierarchy rebuilds.
+- **Conservative detection (5s/30s)**: Rejected — 30 seconds of data collection downtime is too long for SCADA.
+- **Shorter stable-after (10s)**: Rejected — matching the failure threshold risks downing nodes that are slow to respond (GC pause, heavy load).

docs/plans/2026-03-16-communication-layer-refinement-design.md

@@ -0,0 +1,47 @@
+# Communication Layer Refinement — Design
+
+**Date**: 2026-03-16
+**Component**: Central–Site Communication (`Component-Communication.md`)
+**Status**: Approved
+
+## Problem
+
+The Communication Layer doc defined 8 message patterns clearly but lacked specification for timeouts, transport configuration, reconnection behavior, message ordering guarantees, and connection failure handling.
+
+## Decisions
+
+### Message Timeouts
+- **Per-pattern timeouts with sensible defaults**, overridable in configuration.
+- Deployment and system-wide artifacts: 120 seconds (script compilation can be slow).
+- Lifecycle commands, integration routing, recipe/command delivery, remote queries: 30 seconds.
+- Uses the Akka.NET ask pattern; timeout results in failure to caller.
+
+### Transport Configuration
+- **Akka.NET built-in reconnection** with explicitly configured transport heartbeat interval and failure detection threshold.
+- No custom reconnection logic — framework handles it.
+- Settings explicitly documented rather than relying on framework defaults, for predictability in a SCADA context.
+
+### Connection Failure Behavior
+- **In-flight messages get a timeout error** — caller retries manually. No buffering at central. Consistent with existing design principle.
+- Automatic retry rejected due to risk of duplicate processing (e.g., site may have applied a deployment before the connection dropped).
+
+### Message Ordering
+- **Per-site ordering guaranteed** — relies on Akka.NET's built-in per-sender/per-receiver ordering. No custom sequencing logic needed.
+
+### Debug Stream Interruption
+- **Stream dies on any disconnect** (failover or network blip). Engineer reopens the debug view manually.
+- Auto-resume rejected — adds complexity for a transient diagnostic tool.
+
+## Affected Documents
+
+| Document | Change |
+|----------|--------|
+| `Component-Communication.md` | Added 4 new sections: Message Timeouts, Transport Configuration, Message Ordering, Connection Failure Behavior |
+
+## Alternatives Considered
+
+- **Global timeout for all patterns**: Rejected — deployment involves compilation and needs more time than a simple query.
+- **Default Akka.NET transport settings**: Rejected — relying on undocumented defaults is risky for SCADA; explicit configuration ensures predictable behavior.
+- **Automatic retry of in-flight messages**: Rejected — risks duplicate processing and contradicts the no-buffering-at-central principle.
+- **No ordering guarantee**: Rejected — Akka.NET provides this for free; the design already implicitly relies on it.
+- **Auto-resume debug streams on reconnection**: Rejected — adds state tracking complexity for a transient diagnostic feature.