ScadaBridge/docs/components/ClusterInfrastructure.md

Cluster Infrastructure

The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.

Overview

Cluster Infrastructure (#13) is a design responsibility spanning two projects rather than a single buildable project:

• src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ owns the cluster configuration contract: ClusterOptions (seed nodes, failure-detection timings, downing strategy), ClusterOptionsValidator, and the AddClusterInfrastructure DI extension that registers the validator. It does not start an actor system.
• src/ZB.MOM.WW.ScadaBridge.Host/ owns the cluster bootstrap and runtime wiring: AkkaHostedService builds the Akka HOCON from ClusterOptions and NodeOptions, starts the ActorSystem, wires CoordinatedShutdown, and creates all role-specific actors including the cluster singletons.

This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. ClusterInfrastructure is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.

Both central and site clusters run this same topology: two nodes, one active (the oldest Up member), one standby, with automatic failover and no manual intervention required for dual-node recovery.

Key Concepts

One ActorSystem name for every cluster

Every node in every cluster — central and all sites — joins an ActorSystem named "scadabridge", hardcoded at AkkaHostedService.cs:191 (ActorSystem.Create("scadabridge", config)). Central and each site are separate clusters only by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a ClusterClient could not reach a differently-named system.

Active/standby is the oldest Up member — never the cluster leader

A node is "active" when it is the oldest Up member of its role scope — the member ClusterSingletonManager places singletons on. Akka's cluster leader (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads cluster.State.Leader:

• ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role) (Communication/ClusterState/ActiveNodeEvaluator.cs:35) is the single implementation — self is Up, carries the role when one is given, and no other Up member in that scope is older (self.IsOlderThan(m)).
• ClusterActivityEvaluator.SelfIsOldest (Host/Health/ClusterActivityEvaluator.cs:23) delegates to it, and is what ActiveNodeGate.IsActiveNode (Host/Health/ActiveNodeGate.cs:48), OldestNodeActiveHealthCheck, and AkkaClusterNodeProvider.SelfIsPrimary all call.
• SiteCommunicationActor stamps its heartbeat's IsActive from the same evaluator (Communication/Actors/SiteCommunicationActor.cs:517-518).

Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.

Configuration contract vs. bootstrap split

ClusterOptions holds the cluster-wide formation and failure-detection settings. Node-identity settings — remoting hostname/port, role (Central or Site), site identifier, gRPC port — live in NodeOptions (ScadaBridge:Node section), owned by the Host. This split prevents the configuration contract from acquiring a hard dependency on Host-specific concerns.

Singleton hosting

Cluster Infrastructure provides the hosting platform; each singleton is owned and created by the component responsible for it. The Host's RegisterCentralActors and RegisterSiteActorsAsync methods wire every singleton via ClusterSingletonManager and a companion ClusterSingletonProxy so other actors can address it through a stable path regardless of which node currently hosts it.

Architecture

HOCON assembly

AkkaHostedService.BuildHocon constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through QuoteHocon (string escaping) and DurationHocon (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.

The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits akka.extensions (registers DistributedPubSubExtensionProvider), akka.remote.dot-netty.tcp (binds NodeOptions.NodeHostname and NodeOptions.RemotingPort), and akka.remote.transport-failure-detector (heartbeat interval and acceptable-heartbeat-pause from CommunicationOptions.TransportHeartbeatInterval / TransportFailureThreshold).

The downing block is not a fixed stanza — BuildHocon branches on ClusterOptions.SplitBrainResolverStrategy and emits one of two shapes (AkkaHostedService.cs:275-286):

// Abbreviated — see AkkaHostedService.BuildHocon for the full method.
var downingBlock = string.Equals(
    clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
    ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
    : $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
    active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
    stable-after = {DurationHocon(clusterOptions.StableAfter)}
    keep-oldest {{
        down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
    }}
}}";

return $@"
audit-telemetry-dispatcher {{
    type = ForkJoinDispatcher
    throughput = 100
    dedicated-thread-pool {{
        thread-count = 2
    }}
}}
akka {{
    // akka.extensions, akka.remote.dot-netty.tcp, and
    // akka.remote.transport-failure-detector also emitted here (see full method).
    actor {{
        provider = cluster
    }}
    cluster {{
        seed-nodes = [{seedNodesStr}]
        roles = [{rolesStr}]
        min-nr-of-members = {clusterOptions.MinNrOfMembers}
        {downingBlock}
        failure-detector {{
            heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
            acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
        }}
        run-coordinated-shutdown-when-down = on
    }}
    coordinated-shutdown {{
        run-by-clr-shutdown-hook = on
    }}
}}";

A downing-provider-class is always named explicitly. Akka defaults to NoDowning, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.

The HOCON also defines the audit-telemetry-dispatcher (a two-thread ForkJoinDispatcher) so SiteAuditTelemetryActor's SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.

Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no enable-ssl, no require-cookie, no trusted-selection-paths. Akka remoting between nodes and from a ClusterClient is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.

Downing strategy (auto-down — availability-first)

Decision 2026-07-21 (docs/plans/2026-07-21-auto-down-availability-decision.md): the default strategy is auto-down — Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter (15 s). The leader among the reachable members downs the unreachable peer once the stability window elapses.

• Either-node crash is survivable. If the standby crashes, the active node downs it and continues. If the active/oldest node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and /health/active flips to it — no operator action and no victim restart.
• The accepted trade is dual-active during a real network partition. With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
• StableAfter is the debounce, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.

keep-oldest remains a supported value (ClusterOptionsValidator allows exactly auto-down and keep-oldest) for deployments that prefer partition-safety, but it cannot survive a crash of the oldest node in a two-node cluster: Akka's down-if-alone only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes DownReachable and downs itself. Quorum strategies are rejected outright — static-quorum with quorum 1 trips Akka's IsTooManyMembers guard and downs all members on any unreachability, and keep-majority merely moves the fatal crash from the oldest node to the lowest-address node.

Downed-node recovery

run-coordinated-shutdown-when-down = on means a downed node runs CoordinatedShutdown and terminates its own ActorSystem. The Host watches ActorSystem.WhenTerminated; a termination that is not the host's own StopAsync calls IHostApplicationLifetime.StopApplication() so the process exits and the service supervisor (docker restart: unless-stopped, Windows service recovery) restarts it as a fresh incarnation (AkkaHostedService.cs:203-218).

Seed-node ordering (decision 2026-07-22). Only the first seed listed in Cluster:SeedNodes may self-join to form a new cluster — Akka runs FirstSeedNodeProcess for it and JoinSeedNodeProcess (which can never form one) for everyone else. Every node therefore lists itself first and its partner second, so any node can boot alone and become operational unattended; StartupValidator fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on InitJoin until its peer returned — the registered outage gap. See docs/requirements/Component-ClusterInfrastructure.md → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.

Failure detection and failover timeline

Detection uses two independent Akka heartbeat channels:

• Cluster failure detector (akka.cluster.failure-detector): monitors membership, triggers the Unreachable events the downing provider acts on.
• Transport failure detector (akka.remote.transport-failure-detector): monitors the underlying TCP transport between nodes; configured separately from CommunicationOptions.TransportHeartbeatInterval / TransportFailureThreshold.

With the defaults in ClusterOptions, the total failover budget is approximately 25 seconds:

Phase	Duration	Source
Failure detection (acceptable-heartbeat-pause)	10 s	ClusterOptions.FailureDetectionThreshold
Downing window (auto-down-unreachable-after)	15 s	ClusterOptions.StableAfter
Singleton restart	< 1 s	Actor PreStart

The docker failover drill (docker/failover-drill.sh) measures both directions — standby mode kills the younger node, active mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.

Graceful shutdown and singleton handover

When a node is stopped cleanly, CoordinatedShutdown runs before the CLR exits (run-by-clr-shutdown-hook = on). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.

Every singleton is created through the shared SingletonRegistrar.Start helper (Host/Actors/SingletonRegistrar.cs), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical {name}-singleton / {name}-proxy naming, a PoisonPill termination message, an optional .WithRole(role) on both the manager and proxy settings, and a PhaseClusterLeave task that GracefulStops the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:

// SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
    Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
    $"drain-{name}-singleton",
    async () =>
    {
        try
        {
            await manager.GracefulStop(timeout);
        }
        catch (Exception ex)
        {
            logger.LogWarning(ex,
                "{Singleton} singleton did not drain within the graceful-stop timeout; "
                + "falling through to PoisonPill handover", name);
        }
        return Akka.Done.Instance;
    });

Cluster roles and singleton scoping

Each node carries one or more cluster roles set in the HOCON roles list, built by AkkaHostedService.BuildRoles (AkkaHostedService.cs:406-417). Site nodes carry two roles: the base "Site" role plus a site-specific "site-{SiteId}" (a node with SiteId: "site-a" gets "site-site-a"). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the "Central" role.

Dual-node recovery

Because both nodes are configured as seed nodes and each lists itself first, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot alone forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the InitJoin handshake — they split only under a genuine boot-time partition, the same class auto-down already accepts.

Cluster singletons hosted

The Host wires the following singletons through SingletonRegistrar.Start. Cluster Infrastructure provides the ClusterSingletonManager / ClusterSingletonProxy pattern and the drain hook; each singleton's behaviour is documented in the owning component.

Central singletons (oldest Up central node, no role scope):

Singleton name	Actor class	Owner
notification-outbox	NotificationOutboxActor	Notification Outbox (#21)
audit-log-ingest	AuditLogIngestActor	Audit Log (#23)
site-call-audit	SiteCallAuditActor	Site Call Audit (#22)
audit-log-purge	AuditLogPurgeActor	Audit Log (#23)
site-audit-reconciliation	SiteAuditReconciliationActor	Audit Log (#23)
kpi-history-recorder	KpiHistoryRecorderActor	KPI History
pending-deployment-purge	PendingDeploymentPurgeActor	Deployment Manager (#2)

Site singletons (oldest Up node of that site, scoped to the "site-{SiteId}" role):

Singleton name	Actor class	Owner
deployment-manager	DeploymentManagerActor	Site Runtime (#3)
event-log-handler	EventLogHandlerActor	Site Event Logging (#12)

SiteAuditTelemetryActor (Audit Log #23) is not a singleton — it runs on every site node and reads node-local SQLite. It is created directly with ActorOf and bound to the audit-telemetry-dispatcher.

Usage

Registering the configuration contract

Every host calls AddClusterInfrastructure to register ClusterOptionsValidator:

services.AddClusterInfrastructure();

This registers ClusterOptionsValidator as an IValidateOptions<ClusterOptions> singleton. Because the Host binds ClusterOptions with ValidateOnStart, a misconfigured ScadaBridge:Cluster section throws an OptionsValidationException at startup rather than booting into a broken cluster. The validator rejects: a strategy other than auto-down or keep-oldest; MinNrOfMembers != 1; a non-positive StableAfter, HeartbeatInterval or FailureDetectionThreshold; a HeartbeatInterval not below FailureDetectionThreshold; fewer than two seed nodes unless AllowSingleNodeCluster = true; and DownIfAlone = false only when the strategy is keep-oldest (the flag is inert under auto-down, so any value passes there).

Checking active-node status

Components that must run only on the active node resolve IActiveNodeGate (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-Up evaluator — it never inspects cluster leadership:

// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode
{
    get
    {
        var system = _akkaService.ActorSystem;
        if (system == null)
            return false;

        var cluster = Cluster.Get(system);
        return ClusterActivityEvaluator.SelfIsOldest(cluster);
    }
}

This returns false while the actor system is warming up, and SelfIsOldest returns false unless the node has reached MemberStatus.Up — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and OldestNodeActiveHealthCheck backs /health/active off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.

Configuration

ClusterOptions is bound from ScadaBridge:Cluster. NodeOptions is bound from ScadaBridge:Node.

ScadaBridge:Cluster

Key	Type	Default	Description
SeedNodes	List<string>	(required)	Akka seed-node URIs. Must contain at least 2 entries (1 with AllowSingleNodeCluster); both nodes list both themselves and their partner. Only the first entry may self-form a new cluster.
SplitBrainResolverStrategy	string	"auto-down"	"auto-down" or "keep-oldest". Quorum strategies are rejected by ClusterOptionsValidator. See downing strategy above.
StableAfter	TimeSpan	00:00:15	Sustained unreachability before downing. Emitted as auto-down-unreachable-after under auto-down, as the SBR stable-after under keep-oldest.
HeartbeatInterval	TimeSpan	00:00:02	Cluster failure-detector heartbeat frequency. Must be less than FailureDetectionThreshold.
FailureDetectionThreshold	TimeSpan	00:00:10	acceptable-heartbeat-pause for the cluster failure detector.
MinNrOfMembers	int	1	Must be 1. A value of 2 blocks the cluster singleton after failover.
DownIfAlone	bool	true	keep-oldest only — inert under auto-down. Validated as true only when the strategy is keep-oldest.
AllowSingleNodeCluster	bool	false	Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two.

ScadaBridge:Node

Key	Type	Default	Description
Role	string	(required)	"Central" or "Site".
NodeHostname	string	(required)	Hostname this node advertises to the Akka cluster remoting layer.
NodeName	string	""	Semantic label stamped on audit rows (SourceNode). Conventional values: node-a/node-b for sites, central-a/central-b for central.
SiteId	string?	(required for Site)	Site identifier; appended to the site-specific cluster role (site-{SiteId}).
RemotingPort	int	8081	Akka.NET TCP remoting port. Code default is 8081; the site deployment overrides this to 8082 via appsettings.Site.json.
GrpcPort	int	8083	Kestrel HTTP/2 port for SiteStreamGrpcServer (site nodes only). Must differ from RemotingPort.
MetricsPort	int	8084	Kestrel HTTP/1.1 port for the Prometheus /metrics scrape endpoint (site nodes only). Must differ from RemotingPort and GrpcPort.

Representative docker configuration (central node A)

{
  "ScadaBridge": {
    "Node": {
      "Role": "Central",
      "NodeName": "central-a",
      "NodeHostname": "scadabridge-central-a",
      "RemotingPort": 8081
    },
    "Cluster": {
      "SeedNodes": [
        "akka.tcp://scadabridge@scadabridge-central-a:8081",
        "akka.tcp://scadabridge@scadabridge-central-b:8081"
      ],
      "SplitBrainResolverStrategy": "auto-down",
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
      "MinNrOfMembers": 1
    }
  }
}

DownIfAlone is not present in the docker files because it is a keep-oldest-only knob and every shipped deployment runs auto-down, under which the flag is inert.

Dependencies & Interactions

• Host (#15) — owns the Akka.NET bootstrap. AkkaHostedService consumes ClusterOptions and NodeOptions, assembles the HOCON, starts the ActorSystem, creates all role-specific actors, and wires CoordinatedShutdown. The ClusterInfrastructure project has no compile-time dependency on the Host; the dependency is reversed at runtime.
• Commons (#16) — provides INodeIdentityProvider (implemented by NodeIdentityProvider in the Host), which supplies the NodeName label that audit writers stamp on the SourceNode column. Also provides IClusterNodeProvider (implemented by AkkaClusterNodeProvider in the Host), which the Health Monitoring component uses to report per-node up/down status.
• Health Monitoring (#11) — uses IClusterNodeProvider to list cluster members and determine whether the local node is primary; uses IActiveNodeGate (central only) to gate active-node-only health paths. The active/standby distinction reported to central health originates here.
• Site Runtime (#3) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
• Notification Outbox (#21), Site Call Audit (#22), Audit Log (#23) — each hosts one or more central singletons wired by RegisterCentralActors. Cluster Infrastructure provides the ClusterSingletonManager/ClusterSingletonProxy boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
• Central–Site Communication (#5) — CentralCommunicationActor and SiteCommunicationActor are created and registered with ClusterClientReceptionist inside the same AkkaHostedService startup, making them addressable by remote ClusterClient instances. The transport-level heartbeat (TransportHeartbeatInterval, TransportFailureThreshold) is configured separately from the cluster failure-detector and comes from CommunicationOptions.
• Inbound API (#14) — resolves IActiveNodeGate to return HTTP 503 on standby central nodes. Gate returns false until the actor system is Up and this node is the oldest Up member.
• Design spec: Component-ClusterInfrastructure.md.

Troubleshooting

Node fails to join cluster on startup

ClusterOptionsValidator rejects fewer than two seed nodes (without AllowSingleNodeCluster), a strategy outside auto-down / keep-oldest, MinNrOfMembers != 1, or DownIfAlone = false under keep-oldest, at startup with an OptionsValidationException. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, StartupValidator explicitly rejects seed entries whose port matches GrpcPort.

A node that boots, logs no validation error, but never reaches Up was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in SeedNodes and the first seed was down, so it looped on InitJoin waiting for a peer that could form the cluster. Self-first ordering plus the StartupValidator rule that enforces it should make this unreachable; if you still see it, check that seed-nodes[0] really resolves to this node's own NodeHostname:RemotingPort (the validator compares host and port, and Akka does no DNS canonicalisation — node-a and node-a.example.com are different seed identities).

Singleton not starting after failover

If the surviving node is Up but singletons do not start, MinNrOfMembers is the first thing to check. A value of 2 keeps the surviving node waiting for a second member indefinitely. The validator enforces 1, but a manually patched appsettings.json that bypasses the validator could produce this.

Two live clusters (dual-active)

Under auto-down this is the accepted trade, not a misconfiguration: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-Up, both host a full set of singletons, and both answer /health/active with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp IsActive on their heartbeats; the Health dashboard shows two Primaries). The two sides do not merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart one side; it rejoins its peer as a fresh incarnation and comes back as standby.

Deployments that would rather lose availability than run dual-active should set SplitBrainResolverStrategy: "keep-oldest" (with DownIfAlone = true), accepting that a crash of the oldest node is then a total outage.

Graceful shutdown takes longer than expected

If a clean node stop takes up to 25 seconds instead of seconds, CoordinatedShutdown may not be running — check that run-by-clr-shutdown-hook = on is present in the assembled HOCON (it is emitted unconditionally by BuildHocon) and that the Windows Service stop signal reaches the process rather than being killed. A SIGKILL / TerminateProcess bypasses CoordinatedShutdown entirely; the surviving node then has to wait the full failure-detection window.

Related Documentation

• Cluster Infrastructure design specification
• Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)
• Host
• Site Runtime
• Health Monitoring
• Central–Site Communication
• Notification Outbox
• Site Call Audit
• Audit Log
• Commons