ScadaBridge/docs/components/ClusterInfrastructure.md

# Cluster Infrastructure

The Cluster Infrastructure component manages Akka.NET cluster formation, active/standby failover, the downing strategy for unreachable members, and the singleton hosting that all other ScadaBridge components depend on. Every site and central cluster is a two-node active/standby pair governed by the same configuration contract and bootstrap logic.

## Overview

Cluster Infrastructure (#13) is a **design responsibility** spanning two projects rather than a single buildable project:

- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/`** owns the cluster configuration contract: `ClusterOptions` (seed nodes, failure-detection timings, downing strategy), `ClusterOptionsValidator`, and the `AddClusterInfrastructure` DI extension that registers the validator. It does not start an actor system.
- **`src/ZB.MOM.WW.ScadaBridge.Host/`** owns the cluster bootstrap and runtime wiring: `AkkaHostedService` builds the Akka HOCON from `ClusterOptions` and `NodeOptions`, starts the `ActorSystem`, wires `CoordinatedShutdown`, and creates all role-specific actors including the cluster singletons.

This split is deliberate. The Host is the single deployable binary and the only project that performs Akka.NET bootstrap, so all cluster bring-up lives there. `ClusterInfrastructure` is the portable configuration contract that the Host consumes — it can be referenced by tests and other components without pulling in the Host.

Both central and site clusters run this same topology: two nodes, one active (the oldest `Up` member), one standby, with automatic failover and no manual intervention required for dual-node recovery.

## Key Concepts

### One `ActorSystem` name for every cluster

Every node in every cluster — central and all sites — joins an `ActorSystem` named **`"scadabridge"`**, hardcoded at `AkkaHostedService.cs:191` (`ActorSystem.Create("scadabridge", config)`). Central and each site are separate clusters *only* by seed-node partitioning, not by system name. This is required rather than incidental: Akka.Remote matches addresses including the system name, so a `ClusterClient` could not reach a differently-named system.

### Active/standby is the oldest `Up` member — never the cluster leader

A node is "active" when it is the **oldest `Up` member** of its role scope — the member `ClusterSingletonManager` places singletons on. Akka's *cluster leader* (lowest address) is a different, Akka-internal concept: it diverges from singleton placement permanently once the original first node restarts and rejoins. Every product-level active/standby decision therefore goes through one evaluator and never reads `cluster.State.Leader`:

- `ActiveNodeEvaluator.SelfIsOldestUp(Cluster, string? role)` (`Communication/ClusterState/ActiveNodeEvaluator.cs:35`) is the single implementation — self is `Up`, carries the role when one is given, and no other `Up` member in that scope is older (`self.IsOlderThan(m)`).
- `ClusterActivityEvaluator.SelfIsOldest` (`Host/Health/ClusterActivityEvaluator.cs:23`) delegates to it, and is what `ActiveNodeGate.IsActiveNode` (`Host/Health/ActiveNodeGate.cs:48`), `OldestNodeActiveHealthCheck`, and `AkkaClusterNodeProvider.SelfIsPrimary` all call.
- `SiteCommunicationActor` stamps its heartbeat's `IsActive` from the same evaluator (`Communication/Actors/SiteCommunicationActor.cs:517-518`).

Cluster singletons automatically migrate to the surviving node on failover, and because "active" is defined as the singleton-placement member, the health/routing view and the singleton view can never disagree.

### Configuration contract vs. bootstrap split

`ClusterOptions` holds the cluster-wide formation and failure-detection settings. Node-identity settings — remoting hostname/port, role (`Central` or `Site`), site identifier, gRPC port — live in `NodeOptions` (`ScadaBridge:Node` section), owned by the Host. This split prevents the configuration contract from acquiring a hard dependency on Host-specific concerns.

### Singleton hosting

Cluster Infrastructure provides the hosting platform; each singleton is owned and created by the component responsible for it. The Host's `RegisterCentralActors` and `RegisterSiteActorsAsync` methods wire every singleton via `ClusterSingletonManager` and a companion `ClusterSingletonProxy` so other actors can address it through a stable path regardless of which node currently hosts it.

## Architecture

### HOCON assembly

`AkkaHostedService.BuildHocon` constructs the Akka HOCON document from the bound options at startup. All interpolated values pass through `QuoteHocon` (string escaping) and `DurationHocon` (millisecond rendering) so the document is never corrupted by hostnames or timing values containing special characters or sub-second precision.

The snippet below is abbreviated to highlight the cluster stanzas. The full method also emits `akka.extensions` (registers `DistributedPubSubExtensionProvider`), `akka.remote.dot-netty.tcp` (binds `NodeOptions.NodeHostname` and `NodeOptions.RemotingPort`), and `akka.remote.transport-failure-detector` (heartbeat interval and acceptable-heartbeat-pause from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`).

The downing block is **not** a fixed stanza — `BuildHocon` branches on `ClusterOptions.SplitBrainResolverStrategy` and emits one of two shapes (`AkkaHostedService.cs:275-286`):

```csharp
// Abbreviated — see AkkaHostedService.BuildHocon for the full method.
var downingBlock = string.Equals(
    clusterOptions.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase)
    ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
    : $@"downing-provider-class = ""Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster""
split-brain-resolver {{
    active-strategy = {QuoteHocon(clusterOptions.SplitBrainResolverStrategy)}
    stable-after = {DurationHocon(clusterOptions.StableAfter)}
    keep-oldest {{
        down-if-alone = {(clusterOptions.DownIfAlone ? "on" : "off")}
    }}
}}";

return $@"
audit-telemetry-dispatcher {{
    type = ForkJoinDispatcher
    throughput = 100
    dedicated-thread-pool {{
        thread-count = 2
    }}
}}
akka {{
    // akka.extensions, akka.remote.dot-netty.tcp, and
    // akka.remote.transport-failure-detector also emitted here (see full method).
    actor {{
        provider = cluster
    }}
    cluster {{
        seed-nodes = [{seedNodesStr}]
        roles = [{rolesStr}]
        min-nr-of-members = {clusterOptions.MinNrOfMembers}
        {downingBlock}
        failure-detector {{
            heartbeat-interval = {DurationHocon(clusterOptions.HeartbeatInterval)}
            acceptable-heartbeat-pause = {DurationHocon(clusterOptions.FailureDetectionThreshold)}
        }}
        run-coordinated-shutdown-when-down = on
    }}
    coordinated-shutdown {{
        run-by-clr-shutdown-hook = on
    }}
}}";
```

A `downing-provider-class` is always named explicitly. Akka defaults to `NoDowning`, under which the downing configuration is inert and singletons never migrate on a hard crash or partition; naming the provider is what activates automatic downing.

The HOCON also defines the `audit-telemetry-dispatcher` (a two-thread `ForkJoinDispatcher`) so `SiteAuditTelemetryActor`'s SQLite reads and gRPC pushes never contend with the default dispatcher used by hot-path actors.

Nothing in the emitted document enables remoting TLS or an Akka secure cookie — there is no `enable-ssl`, no `require-cookie`, no `trusted-selection-paths`. Akka remoting between nodes and from a `ClusterClient` is plaintext and unauthenticated; the deployment is assumed to sit on a trusted network.

### Downing strategy (auto-down — availability-first)

**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.

- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **`StableAfter` is the debounce**, not a resolver phase: 15 s of sustained unreachability before downing, which absorbs startup, rolling restarts, and transient blips.

`keep-oldest` remains a supported value (`ClusterOptionsValidator` allows exactly `auto-down` and `keep-oldest`) for deployments that prefer partition-safety, but it **cannot survive a crash of the oldest node in a two-node cluster**: Akka's `down-if-alone` only rescues the survivor when its own side has ≥ 2 members, so a 1-vs-1 survivor takes `DownReachable` and downs *itself*. Quorum strategies are rejected outright — `static-quorum` with quorum 1 trips Akka's `IsTooManyMembers` guard and downs *all* members on any unreachability, and `keep-majority` merely moves the fatal crash from the oldest node to the lowest-address node.

### Downed-node recovery

`run-coordinated-shutdown-when-down = on` means a downed node runs `CoordinatedShutdown` and terminates its own `ActorSystem`. The Host watches `ActorSystem.WhenTerminated`; a termination that is not the host's own `StopAsync` calls `IHostApplicationLifetime.StopApplication()` so the process exits and the service supervisor (docker `restart: unless-stopped`, Windows service recovery) restarts it as a fresh incarnation (`AkkaHostedService.cs:203-218`).

**Seed-node ordering (decision 2026-07-22).** Only the *first* seed listed in `Cluster:SeedNodes` may self-join to form a new cluster — Akka runs `FirstSeedNodeProcess` for it and `JoinSeedNodeProcess` (which can never form one) for everyone else. Every node therefore lists **itself** first and its partner second, so any node can boot alone and become operational unattended; `StartupValidator` fails the boot if that ordering is broken. Until this change all nodes shared one first seed, and a node that had to boot alone looped on `InitJoin` until its peer returned — the registered outage gap. See `docs/requirements/Component-ClusterInfrastructure.md` → Seed Node Ordering for the scenario table and for why an external self-form timer was rejected.

### Failure detection and failover timeline

Detection uses two independent Akka heartbeat channels:

- **Cluster failure detector** (`akka.cluster.failure-detector`): monitors membership, triggers the `Unreachable` events the downing provider acts on.
- **Transport failure detector** (`akka.remote.transport-failure-detector`): monitors the underlying TCP transport between nodes; configured separately from `CommunicationOptions.TransportHeartbeatInterval` / `TransportFailureThreshold`.

With the defaults in `ClusterOptions`, the total failover budget is approximately 25 seconds:

| Phase | Duration | Source |
|-------|----------|--------|
| Failure detection (`acceptable-heartbeat-pause`) | 10 s | `ClusterOptions.FailureDetectionThreshold` |
| Downing window (`auto-down-unreachable-after`) | 15 s | `ClusterOptions.StableAfter` |
| Singleton restart | < 1 s | Actor `PreStart` |

The docker failover drill (`docker/failover-drill.sh`) measures both directions — `standby` mode kills the younger node, `active` mode kills the active/oldest node and asserts the survivor takes over while the victim is still down.

### Graceful shutdown and singleton handover

When a node is stopped cleanly, `CoordinatedShutdown` runs before the CLR exits (`run-by-clr-shutdown-hook = on`). The cluster-leave phase signals Akka to migrate singletons before the actor system terminates, so handover happens in seconds rather than waiting for the full failure-detection timeout.

Every singleton is created through the shared `SingletonRegistrar.Start` helper (`Host/Actors/SingletonRegistrar.cs`), so the drain is uniform rather than per-singleton boilerplate. The registrar applies the canonical `{name}-singleton` / `{name}-proxy` naming, a `PoisonPill` termination message, an optional `.WithRole(role)` on both the manager and proxy settings, and a `PhaseClusterLeave` task that `GracefulStop`s the manager (10-second default) so in-flight EF Core (central) or SQLite (site) work completes before handover opens:

```csharp
// SingletonRegistrar.Start — the drain task registered for every singleton
Akka.Actor.CoordinatedShutdown.Get(system).AddTask(
    Akka.Actor.CoordinatedShutdown.PhaseClusterLeave,
    $"drain-{name}-singleton",
    async () =>
    {
        try
        {
            await manager.GracefulStop(timeout);
        }
        catch (Exception ex)
        {
            logger.LogWarning(ex,
                "{Singleton} singleton did not drain within the graceful-stop timeout; "
                + "falling through to PoisonPill handover", name);
        }
        return Akka.Done.Instance;
    });
```

### Cluster roles and singleton scoping

Each node carries one or more cluster roles set in the HOCON `roles` list, built by `AkkaHostedService.BuildRoles` (`AkkaHostedService.cs:406-417`). Site nodes carry **two** roles: the base `"Site"` role plus a site-specific `"site-{SiteId}"` (a node with `SiteId: "site-a"` gets `"site-site-a"`). Singletons on site clusters are scoped to the site-specific role so each site's singleton runs on exactly one node of that site's cluster. Central singletons pass no role to the registrar and so are unscoped — all central nodes share the `"Central"` role.

### Dual-node recovery

Because both nodes are configured as seed nodes **and each lists itself first**, whichever node starts first after a simultaneous failure forms a new cluster; the second joins when it comes up. There is no pre-existing cluster to conflict with, so the "both starting fresh" case needs no downing decision at all. Since 2026-07-22 there is no remaining ordering dependency: a node that must boot *alone* forms a cluster regardless of which node it is. Two nodes cold-starting at the same moment converge on one cluster via the `InitJoin` handshake — they split only under a genuine boot-time partition, the same class `auto-down` already accepts.

### Cluster singletons hosted

The Host wires the following singletons through `SingletonRegistrar.Start`. Cluster Infrastructure provides the `ClusterSingletonManager` / `ClusterSingletonProxy` pattern and the drain hook; each singleton's behaviour is documented in the owning component.

**Central singletons (oldest `Up` central node, no role scope):**

| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
| `notification-outbox` | `NotificationOutboxActor` | Notification Outbox (#21) |
| `audit-log-ingest` | `AuditLogIngestActor` | Audit Log (#23) |
| `site-call-audit` | `SiteCallAuditActor` | Site Call Audit (#22) |
| `audit-log-purge` | `AuditLogPurgeActor` | Audit Log (#23) |
| `site-audit-reconciliation` | `SiteAuditReconciliationActor` | Audit Log (#23) |
| `kpi-history-recorder` | `KpiHistoryRecorderActor` | KPI History |
| `pending-deployment-purge` | `PendingDeploymentPurgeActor` | Deployment Manager (#2) |

**Site singletons (oldest `Up` node of that site, scoped to the `"site-{SiteId}"` role):**

| Singleton name | Actor class | Owner |
|----------------|-------------|-------|
| `deployment-manager` | `DeploymentManagerActor` | Site Runtime (#3) |
| `event-log-handler` | `EventLogHandlerActor` | Site Event Logging (#12) |

`SiteAuditTelemetryActor` (Audit Log #23) is **not** a singleton — it runs on every site node and reads node-local SQLite. It is created directly with `ActorOf` and bound to the `audit-telemetry-dispatcher`.

## Usage

### Registering the configuration contract

Every host calls `AddClusterInfrastructure` to register `ClusterOptionsValidator`:

```csharp
services.AddClusterInfrastructure();
```

This registers `ClusterOptionsValidator` as an `IValidateOptions<ClusterOptions>` singleton. Because the Host binds `ClusterOptions` with `ValidateOnStart`, a misconfigured `ScadaBridge:Cluster` section throws an `OptionsValidationException` at startup rather than booting into a broken cluster. The validator rejects: a strategy other than `auto-down` or `keep-oldest`; `MinNrOfMembers != 1`; a non-positive `StableAfter`, `HeartbeatInterval` or `FailureDetectionThreshold`; a `HeartbeatInterval` not below `FailureDetectionThreshold`; fewer than two seed nodes unless `AllowSingleNodeCluster = true`; and `DownIfAlone = false` **only when the strategy is `keep-oldest`** (the flag is inert under `auto-down`, so any value passes there).

### Checking active-node status

Components that must run only on the active node resolve `IActiveNodeGate` (registered by the Host's Central composition root). The gate is a thin wrapper over the oldest-`Up` evaluator — it never inspects cluster leadership:

```csharp
// Host/Health/ActiveNodeGate.cs
public bool IsActiveNode
{
    get
    {
        var system = _akkaService.ActorSystem;
        if (system == null)
            return false;

        var cluster = Cluster.Get(system);
        return ClusterActivityEvaluator.SelfIsOldest(cluster);
    }
}
```

This returns `false` while the actor system is warming up, and `SelfIsOldest` returns `false` unless the node has reached `MemberStatus.Up` — the safe-by-default answer matching the standby case. The Inbound API uses this gate to return HTTP 503 on standby nodes, and `OldestNodeActiveHealthCheck` backs `/health/active` off the same evaluator, so the proxy's routing decision and the API's gating decision can never disagree.

## Configuration

`ClusterOptions` is bound from `ScadaBridge:Cluster`. `NodeOptions` is bound from `ScadaBridge:Node`.

### `ScadaBridge:Cluster`

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `SeedNodes` | `List<string>` | (required) | Akka seed-node URIs. Must contain at least 2 entries (1 with `AllowSingleNodeCluster`); both nodes list both themselves and their partner. Only the **first** entry may self-form a new cluster. |
| `SplitBrainResolverStrategy` | `string` | `"auto-down"` | `"auto-down"` or `"keep-oldest"`. Quorum strategies are rejected by `ClusterOptionsValidator`. See downing strategy above. |
| `StableAfter` | `TimeSpan` | `00:00:15` | Sustained unreachability before downing. Emitted as `auto-down-unreachable-after` under `auto-down`, as the SBR `stable-after` under `keep-oldest`. |
| `HeartbeatInterval` | `TimeSpan` | `00:00:02` | Cluster failure-detector heartbeat frequency. Must be less than `FailureDetectionThreshold`. |
| `FailureDetectionThreshold` | `TimeSpan` | `00:00:10` | `acceptable-heartbeat-pause` for the cluster failure detector. |
| `MinNrOfMembers` | `int` | `1` | Must be `1`. A value of `2` blocks the cluster singleton after failover. |
| `DownIfAlone` | `bool` | `true` | `keep-oldest` only — inert under `auto-down`. Validated as `true` only when the strategy is `keep-oldest`. |
| `AllowSingleNodeCluster` | `bool` | `false` | Acknowledges a deliberate single-node install: permits exactly one seed node instead of the usual two. |

### `ScadaBridge:Node`

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| `Role` | `string` | (required) | `"Central"` or `"Site"`. |
| `NodeHostname` | `string` | (required) | Hostname this node advertises to the Akka cluster remoting layer. |
| `NodeName` | `string` | `""` | Semantic label stamped on audit rows (`SourceNode`). Conventional values: `node-a`/`node-b` for sites, `central-a`/`central-b` for central. |
| `SiteId` | `string?` | (required for Site) | Site identifier; appended to the site-specific cluster role (`site-{SiteId}`). |
| `RemotingPort` | `int` | `8081` | Akka.NET TCP remoting port. Code default is `8081`; the site deployment overrides this to `8082` via `appsettings.Site.json`. |
| `GrpcPort` | `int` | `8083` | Kestrel HTTP/2 port for `SiteStreamGrpcServer` (site nodes only). Must differ from `RemotingPort`. |
| `MetricsPort` | `int` | `8084` | Kestrel HTTP/1.1 port for the Prometheus `/metrics` scrape endpoint (site nodes only). Must differ from `RemotingPort` and `GrpcPort`. |

### Representative docker configuration (central node A)

```json
{
  "ScadaBridge": {
    "Node": {
      "Role": "Central",
      "NodeName": "central-a",
      "NodeHostname": "scadabridge-central-a",
      "RemotingPort": 8081
    },
    "Cluster": {
      "SeedNodes": [
        "akka.tcp://scadabridge@scadabridge-central-a:8081",
        "akka.tcp://scadabridge@scadabridge-central-b:8081"
      ],
      "SplitBrainResolverStrategy": "auto-down",
      "StableAfter": "00:00:15",
      "HeartbeatInterval": "00:00:02",
      "FailureDetectionThreshold": "00:00:10",
      "MinNrOfMembers": 1
    }
  }
}
```

`DownIfAlone` is not present in the docker files because it is a `keep-oldest`-only knob and every shipped deployment runs `auto-down`, under which the flag is inert.

## Dependencies & Interactions

- [Host (#15)](./Host.md) — owns the Akka.NET bootstrap. `AkkaHostedService` consumes `ClusterOptions` and `NodeOptions`, assembles the HOCON, starts the `ActorSystem`, creates all role-specific actors, and wires `CoordinatedShutdown`. The `ClusterInfrastructure` project has no compile-time dependency on the Host; the dependency is reversed at runtime.
- [Commons (#16)](./Commons.md) — provides `INodeIdentityProvider` (implemented by `NodeIdentityProvider` in the Host), which supplies the `NodeName` label that audit writers stamp on the `SourceNode` column. Also provides `IClusterNodeProvider` (implemented by `AkkaClusterNodeProvider` in the Host), which the Health Monitoring component uses to report per-node up/down status.
- [Health Monitoring (#11)](./HealthMonitoring.md) — uses `IClusterNodeProvider` to list cluster members and determine whether the local node is primary; uses `IActiveNodeGate` (central only) to gate active-node-only health paths. The active/standby distinction reported to central health originates here.
- [Site Runtime (#3)](./SiteRuntime.md) — the Deployment Manager singleton is the most operationally critical singleton this infrastructure hosts. It re-creates the full Instance Actor hierarchy from local SQLite on failover. Staggered Instance Actor startup after failover is Site Runtime's responsibility; this component provides the singleton placement guarantee.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md) — each hosts one or more central singletons wired by `RegisterCentralActors`. Cluster Infrastructure provides the `ClusterSingletonManager`/`ClusterSingletonProxy` boilerplate and the graceful-shutdown hooks; the business logic lives in the owning component.
- [Central–Site Communication (#5)](./Communication.md) — `CentralCommunicationActor` and `SiteCommunicationActor` are created and registered with `ClusterClientReceptionist` inside the same `AkkaHostedService` startup, making them addressable by remote `ClusterClient` instances. The transport-level heartbeat (`TransportHeartbeatInterval`, `TransportFailureThreshold`) is configured separately from the cluster failure-detector and comes from `CommunicationOptions`.
- [Inbound API (#14)](./InboundAPI.md) — resolves `IActiveNodeGate` to return HTTP 503 on standby central nodes. Gate returns `false` until the actor system is `Up` and this node is the oldest `Up` member.
- Design spec: [Component-ClusterInfrastructure.md](../requirements/Component-ClusterInfrastructure.md).

## Troubleshooting

### Node fails to join cluster on startup

`ClusterOptionsValidator` rejects fewer than two seed nodes (without `AllowSingleNodeCluster`), a strategy outside `auto-down` / `keep-oldest`, `MinNrOfMembers != 1`, or `DownIfAlone = false` under `keep-oldest`, at startup with an `OptionsValidationException`. Check that both seed-node URIs reference the Akka remoting port, not the gRPC port (8083) or metrics port (8084) — on site nodes, `StartupValidator` explicitly rejects seed entries whose port matches `GrpcPort`.

A node that boots, logs no validation error, but never reaches `Up` was — before 2026-07-22 — usually hitting the seed-node bootstrap constraint: it was not the first entry in `SeedNodes` and the first seed was down, so it looped on `InitJoin` waiting for a peer that could form the cluster. Self-first ordering plus the `StartupValidator` rule that enforces it should make this unreachable; if you still see it, check that `seed-nodes[0]` really resolves to this node's own `NodeHostname:RemotingPort` (the validator compares host *and* port, and Akka does no DNS canonicalisation — `node-a` and `node-a.example.com` are different seed identities).

### Singleton not starting after failover

If the surviving node is `Up` but singletons do not start, `MinNrOfMembers` is the first thing to check. A value of `2` keeps the surviving node waiting for a second member indefinitely. The validator enforces `1`, but a manually patched `appsettings.json` that bypasses the validator could produce this.

### Two live clusters (dual-active)

Under `auto-down` this is the **accepted trade, not a misconfiguration**: during a real network partition each side downs the other and continues as a one-node cluster, so both nodes are oldest-`Up`, both host a full set of singletons, and both answer `/health/active` with 200 — including dual MS SQL writers on central. Monitoring surfaces it directly (both nodes stamp `IsActive` on their heartbeats; the Health dashboard shows two Primaries). The two sides do **not** merge on their own — the mutual downing quarantines the association. Recovery is operator-driven: once the link is restored, restart **one** side; it rejoins its peer as a fresh incarnation and comes back as standby.

Deployments that would rather lose availability than run dual-active should set `SplitBrainResolverStrategy: "keep-oldest"` (with `DownIfAlone = true`), accepting that a crash of the oldest node is then a total outage.

### Graceful shutdown takes longer than expected

If a clean node stop takes up to 25 seconds instead of seconds, `CoordinatedShutdown` may not be running — check that `run-by-clr-shutdown-hook = on` is present in the assembled HOCON (it is emitted unconditionally by `BuildHocon`) and that the Windows Service stop signal reaches the process rather than being killed. A `SIGKILL` / `TerminateProcess` bypasses `CoordinatedShutdown` entirely; the surviving node then has to wait the full failure-detection window.

## Related Documentation

- [Cluster Infrastructure design specification](../requirements/Component-ClusterInfrastructure.md)
- [Auto-down downing strategy — availability over partition-safety (decision, 2026-07-21)](../plans/2026-07-21-auto-down-availability-decision.md)
- [Host](./Host.md)
- [Site Runtime](./SiteRuntime.md)
- [Health Monitoring](./HealthMonitoring.md)
- [Central–Site Communication](./Communication.md)
- [Notification Outbox](./NotificationOutbox.md)
- [Site Call Audit](./SiteCallAudit.md)
- [Audit Log](./AuditLog.md)
- [Commons](./Commons.md)