ScadaBridge/docs/requirements/Component-ClusterInfrastructure.md

Component: Cluster Infrastructure

Purpose

The Cluster Infrastructure component manages the Akka.NET cluster setup, active/standby node roles, failover detection, and the foundational runtime environment on which all other components run. It provides the base layer for both central and site clusters.

Location

Both central and site clusters.

Responsibilities

• Bootstrap the Akka.NET actor system on each node.
• Form a two-node cluster (active/standby) using Akka.NET Cluster.
• Manage leader election and role assignment (active vs. standby).
• Detect node failures and trigger failover.
• Provide the Akka.NET remoting infrastructure for inter-cluster communication.
• Support cluster singleton hosting (used by the Site Runtime Deployment Manager singleton on site clusters).
• Manage Windows service lifecycle (start, stop, restart) on each node.

Implementation Note — Code Placement

This component is a design responsibility, not a single buildable project that contains all of the code. The cluster-infrastructure responsibilities above are realised across two projects:

• src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure owns the cluster configuration model: the ClusterOptions POCO (seed nodes, roles, remoting/gRPC ports, failure-detection timings, split-brain settings) bound from appsettings.json via the Options pattern.
• src/ZB.MOM.WW.ScadaBridge.Host owns the cluster bootstrap and runtime wiring: it builds the Akka.NET HOCON from ClusterOptions, starts the ActorSystem, configures the keep-oldest split-brain resolver (down-if-alone = on), wires CoordinatedShutdown into the service lifecycle, and provides active-node / cluster-membership health checks. See Component-Host.md (REQ-HOST-*) for detail.

This split is deliberate — the Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so the cluster bring-up lives there alongside role-based component registration. The ClusterInfrastructure project remains the home of the configuration contract that the Host consumes.

Cluster Topology

Central Cluster

• Two nodes forming an Akka.NET cluster.
• One active node runs all central components (Template Engine, Deployment Manager, Central UI, etc.).
• One standby node is ready to take over on failover.
• Connected to MS SQL databases (Config DB, Machine Data DB).

Site Cluster (per site)

• Two nodes forming an Akka.NET cluster.
• One active node runs all site components (Site Runtime, Data Connection Layer, Store-and-Forward Engine, etc.).
• The Site Runtime Deployment Manager runs as an Akka.NET cluster singleton on the active node, owning the full Instance Actor hierarchy.
• One standby node receives replicated store-and-forward data and is ready to take over.
• Connected to local SQLite databases (store-and-forward buffer, event logs, deployed configurations).
• Connected to machines via data connections (OPC UA).

Cluster Singletons

Akka.NET cluster singletons run on the active node of their cluster and migrate on failover. Each singleton listed here is owned by the named component; this component (Cluster Infrastructure) provides only the hosting, supervision, and active-node placement guarantee.

Central singletons (active central node)

• NotificationOutboxActor — owned by Notification Outbox (#21). Drives the central notification dispatch loop against the Notifications table.
• SiteCallAuditActor — owned by Site Call Audit (#22). Owns the operational SiteCalls table: drives periodic reconciliation pulls for CachedCall / CachedWrite lifecycle, computes KPIs, and relays operator Retry/Discard actions to the owning site. Note: ingest of cached-call telemetry is performed by AuditLogIngestActor (#23) in one transaction with the immutable AuditLog insert — see Component-AuditLog.md, Cached Operations — Combined Telemetry.
• AuditLogIngestActor — owned by Audit Log (#23). Receives gRPC telemetry batches of AuditEvent rows from sites and performs insert-if-not-exists on EventId against the central AuditLog table. For cached-call telemetry (which carries both audit-row content and operational-state fields in a single packet), the ingest performs the AuditLog insert and the SiteCalls upsert in one transaction — see Component-AuditLog.md for the combined-telemetry contract.
• SiteAuditReconciliationActor — owned by Audit Log (#23). Periodic per-site pull (default every 5 minutes) that self-heals missed audit telemetry by asking each site for its oldest ForwardState = 'Pending' row and issuing a PullAuditEvents(sinceUtc, batchSize) when a non-draining backlog is detected.
• AuditLogPurgeActor — owned by Audit Log (#23). Daily partition-switch purge against ps_AuditLog_Month; switches out any partition older than AuditLog:RetentionDays and emits an AuditLog:Purged event. Also rolls the partition scheme forward each month so the next month's partition exists ahead of time.

Site singletons (active site node, per site cluster)

• Site Runtime Deployment Manager — owned by Site Runtime (#3). Owns the full Instance Actor hierarchy; re-creates it on failover from local SQLite.
• SiteAuditTelemetryActor — owned by Audit Log (#23). Drains the local site AuditLog SQLite's ForwardState = 'Pending' rows to central in batches via the existing SiteStream gRPC channel; cadence is short (default 5 s) when the queue is non-empty and longer (default 30 s) when idle. Runs on a dedicated dispatcher so it does not compete with the script blocking-I/O dispatcher (per Component-AuditLog.md, Ingestion Paths → Telemetry forward).

Failover Behavior

Detection

• Akka.NET Cluster monitors node health via heartbeat.
• If the active node becomes unreachable, the standby node detects the failure and promotes itself to active.

Central Failover

• The new active node takes over all central responsibilities.
• In-progress deployments are treated as failed — engineers must retry.
• The UI session may be interrupted — users reconnect to the new active node.
• No message buffering at central — no state to recover beyond what's in MS SQL.

Site Failover

• The new active node takes over:
  • The Deployment Manager singleton restarts and re-creates the full Instance Actor hierarchy by reading deployed configurations from local SQLite. Each Instance Actor spawns its child Script and Alarm Actors.
  • Data collection (Data Connection Layer re-establishes subscriptions as Instance Actors register their data source references).
  • Store-and-forward delivery (buffer is already replicated locally).
• Active debug view streams from central are interrupted — the engineer must re-open them.
• Health reporting resumes from the new active node.
• Alarm states are re-evaluated from incoming values (alarm state is in-memory only).

Downing Strategy (auto-down — availability-first)

Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question): the clusters run the auto-down downing strategy — Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter (15s). The leader among the reachable members downs the unreachable peer after the stability window:

• Either-node crash is survivable. If the standby crashes, the active node downs it and continues (as before). If the active/oldest node crashes, the younger survivor becomes leader among the reachable members, downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and /health/active flips to it — no operator action and no victim restart required. This closes the keep-oldest total-outage gap.
• The accepted trade: dual-active during a real network partition. With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
• Stable-after duration: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.

Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)

• keep-oldest (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the oldest is a total outage. KeepOldest.OldestDecision only lets down-if-alone rescue the survivor when the surviving side has ≥ 2 members (otherSide == 1 && thisSide >= 2); with 1-vs-1 the younger survivor takes DownReachable — it downs itself. Verified live on the docker rig: the survivor logged SBR took decision Akka.Cluster.SBR.DownReachable … including myself, exited, and looped on InitJoin. The strategy remains supported in ClusterOptions (SplitBrainResolverStrategy: keep-oldest) for deployments that prefer partition-safety over availability.
• static-quorum (quorum-size 1): Akka's IsTooManyMembers guard (members > quorum*2-1, i.e. 2 > 1) returns DownAll on any unreachability — total shutdown, strictly worse.
• keep-majority: a 1-vs-1 split keeps the side with the lowest address, which just moves the fatal crash from "the oldest" to "the lowest-address node".
• lease-majority: needs a shared lease store (Kubernetes API, SQL, …) reachable from both nodes — not available at sites (one node per VM, no shared infrastructure).

Downed-node recovery

When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), run-coordinated-shutdown-when-down = on runs CoordinatedShutdown and terminates that node's ActorSystem. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:

1. Down ⇒ CoordinatedShutdown ⇒ ActorSystem termination.
2. The Host watches ActorSystem.WhenTerminated; a termination that is not the host's own StopAsync triggers IHostApplicationLifetime.StopApplication(), so the process exits.
3. The service supervisor restarts it — docker restart: unless-stopped, or Windows service recovery actions (sc.exe failure … restart/…).
4. The restarted process rejoins as a fresh incarnation — but only while a peer still holding cluster state is reachable. A lone restarted node that is not the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.

Seed-node bootstrap constraint (still applies). Only the FIRST seed listed in Cluster:SeedNodes may self-join to form a new cluster. All nodes list the same first seed (e.g. scadabridge-central-a), so a lone restarted non-first-seed node (with the first seed still down) loops on InitJoin forever — never Up, never routable. Under auto-down this constraint no longer causes the active-crash outage (the survivor keeps running and never restarts), but it still bites when a node must boot alone — e.g. a cold start of only the non-first-seed VM, or the survivor itself crashing while its peer is still dead. Recovery is operator-driven — either restart the first-seed node (preferred) or restart the survivor with a self-first seed override (ScadaBridge__Cluster__SeedNodes__0 = self, __1 = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge.

The docker failover drill (docker/failover-drill.sh) proves both directions: standby mode kills the younger node (active untouched, zero routing blips); active mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.

Single-Node Operation

akka.cluster.min-nr-of-members must be set to 1. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.

Failure Detection Timing

Configurable defaults for heartbeat and failure detection:

Setting	Default	Description
Heartbeat interval	2 seconds	Frequency of health check messages between nodes
Failure detection threshold	10 seconds	Time without heartbeat before a node is considered unreachable
Stable-after (split-brain)	15 seconds	Time cluster must be stable before resolver acts
Total failover time	~25 seconds	Detection (10s) + stable-after (15s) + singleton restart

These values balance failover speed with stability — fast enough that data collection gaps are small, tolerant enough that brief network hiccups don't trigger unnecessary failovers.

Dual-Node Recovery

If both nodes in a cluster fail simultaneously (e.g., site power outage):

1. No manual intervention required. Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster. The second node joins when it starts.
2. State recovery (each node has its own local copy of all required data):
   • Site clusters: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
   • Central cluster: All state is in MS SQL (configuration database). The active node resumes normal operation.
3. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.

Graceful Shutdown & Singleton Handover

When a node is stopped for planned maintenance (Windows Service stop), CoordinatedShutdown triggers a graceful leave from the cluster. This enables the Cluster Singleton (Site Runtime Deployment Manager) to hand over to the other node in seconds (limited by the hand-over retry interval) rather than waiting for the full failure detection timeout (~25 seconds).

Configuration required:

• akka.coordinated-shutdown.run-by-clr-shutdown-hook = on
• akka.cluster.run-coordinated-shutdown-when-down = on

The Host component wires CoordinatedShutdown into the Windows Service lifecycle (see REQ-HOST-9).

Node Configuration

Each node is configured with:

• Cluster seed nodes: Both nodes are seed nodes — each node lists both itself and its partner. Either node can start first and form the cluster; the other joins when it starts. No startup ordering dependency.
• Cluster role: Central or Site (plus site identifier for site clusters).
• Akka.NET remoting: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site).
• gRPC port (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport.
• Local storage paths: SQLite database locations (site nodes only).

Windows Service

• Each node runs as a Windows service for automatic startup and recovery.
• Service configuration includes Akka.NET cluster settings and component-specific configuration.

Platform

• OS: Windows Server.
• Runtime: .NET (Akka.NET).
• Cluster: Akka.NET Cluster (application-level, not Windows Server Failover Clustering).

Dependencies

• Akka.NET: Core actor system, cluster, remoting, and cluster singleton libraries.
• Windows: Service hosting, networking.
• MS SQL (central only): Database connectivity.
• SQLite (sites only): Local storage.

Interactions

• All components: Every component runs within the Akka.NET actor system managed by this infrastructure.
• Site Runtime: The Deployment Manager singleton relies on Akka.NET cluster singleton support provided by this infrastructure.
• Communication Layer: Built on top of the Akka.NET remoting provided here.
• Health Monitoring: Reports node status (active/standby) as a health metric.