# Component: Cluster Infrastructure

## Purpose

The Cluster Infrastructure component manages the Akka.NET cluster setup, active/standby node roles, failover detection, and the foundational runtime environment on which all other components run. It provides the base layer for both central and site clusters.

## Location

Both central and site clusters.

## Responsibilities

- Bootstrap the Akka.NET actor system on each node.
- Form a two-node cluster (active/standby) using Akka.NET Cluster.
- Manage leader election and role assignment (active vs. standby).
- Detect node failures and trigger failover.
- Provide the Akka.NET remoting infrastructure for inter-cluster communication.
- Support cluster singleton hosting (used by the Site Runtime Deployment Manager singleton on site clusters).
- Manage Windows service lifecycle (start, stop, restart) on each node.

## Implementation Note — Code Placement

This component is a **design responsibility**, not a single buildable project that
contains all of the code. The cluster-infrastructure responsibilities above are
realised across two projects:

- **`src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure`** owns the cluster **configuration model**:
  the `ClusterOptions` POCO (seed nodes, roles, remoting/gRPC ports, failure-detection
  timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
  builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
  configures the keep-oldest split-brain resolver (`down-if-alone = on`), wires
  `CoordinatedShutdown` into the service lifecycle, and provides active-node /
  cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.

This split is deliberate — the Host is the single deployable binary and the only
project that performs Akka.NET bootstrap, so the cluster bring-up lives there
alongside role-based component registration. The `ClusterInfrastructure` project
remains the home of the configuration contract that the Host consumes.

## Cluster Topology

### Central Cluster
- Two nodes forming an Akka.NET cluster.
- One active node runs all central components (Template Engine, Deployment Manager, Central UI, etc.).
- One standby node is ready to take over on failover.
- Connected to MS SQL databases (Config DB, Machine Data DB).

### Site Cluster (per site)
- Two nodes forming an Akka.NET cluster.
- One active node runs all site components (Site Runtime, Data Connection Layer, Store-and-Forward Engine, etc.).
- The Site Runtime Deployment Manager runs as an **Akka.NET cluster singleton** on the active node, owning the full Instance Actor hierarchy.
- One standby node receives replicated store-and-forward data and is ready to take over.
- Connected to local SQLite databases (store-and-forward buffer, event logs, deployed configurations).
- Connected to machines via data connections (OPC UA).

## Cluster Singletons

Akka.NET cluster singletons run on the active node of their cluster and migrate on failover. Each singleton listed here is owned by the named component; this component (Cluster Infrastructure) provides only the hosting, supervision, and active-node placement guarantee.

### Central singletons (active central node)

- **`NotificationOutboxActor`** — owned by Notification Outbox (#21). Drives the central notification dispatch loop against the `Notifications` table.
- **`SiteCallAuditActor`** — owned by Site Call Audit (#22). Owns the operational `SiteCalls` table: drives periodic reconciliation pulls for `CachedCall` / `CachedWrite` lifecycle, computes KPIs, and relays operator Retry/Discard actions to the owning site. Note: ingest of cached-call telemetry is performed by `AuditLogIngestActor` (#23) in one transaction with the immutable `AuditLog` insert — see Component-AuditLog.md, Cached Operations — Combined Telemetry.
- **`AuditLogIngestActor`** — owned by Audit Log (#23). Receives gRPC telemetry batches of `AuditEvent` rows from sites and performs insert-if-not-exists on `EventId` against the central `AuditLog` table. For cached-call telemetry (which carries both audit-row content and operational-state fields in a single packet), the ingest performs the `AuditLog` insert and the `SiteCalls` upsert in **one transaction** — see Component-AuditLog.md for the combined-telemetry contract.
- **`SiteAuditReconciliationActor`** — owned by Audit Log (#23). Periodic per-site pull (default every 5 minutes) that self-heals missed audit telemetry by asking each site for its oldest `ForwardState = 'Pending'` row and issuing a `PullAuditEvents(sinceUtc, batchSize)` when a non-draining backlog is detected.
- **`AuditLogPurgeActor`** — owned by Audit Log (#23). Daily partition-switch purge against `ps_AuditLog_Month`; switches out any partition older than `AuditLog:RetentionDays` and emits an `AuditLog:Purged` event. Also rolls the partition scheme forward each month so the next month's partition exists ahead of time.

### Site singletons (active site node, per site cluster)

- **Site Runtime Deployment Manager** — owned by Site Runtime (#3). Owns the full Instance Actor hierarchy; re-creates it on failover from local SQLite.
- **`SiteAuditTelemetryActor`** — owned by Audit Log (#23). Drains the local site `AuditLog` SQLite's `ForwardState = 'Pending'` rows to central in batches via the existing `SiteStream` gRPC channel; cadence is short (default 5 s) when the queue is non-empty and longer (default 30 s) when idle. Runs on a **dedicated dispatcher** so it does not compete with the script blocking-I/O dispatcher (per Component-AuditLog.md, Ingestion Paths → Telemetry forward).

## Failover Behavior

### Detection
- Akka.NET Cluster monitors node health via heartbeat.
- If the active node becomes unreachable, the standby node detects the failure and promotes itself to active.

### Central Failover
- The new active node takes over all central responsibilities.
- In-progress deployments are treated as **failed** — engineers must retry.
- The UI session may be interrupted — users reconnect to the new active node.
- No message buffering at central — no state to recover beyond what's in MS SQL.

### Site Failover
- The new active node takes over:
  - The Deployment Manager singleton restarts and re-creates the full Instance Actor hierarchy by reading deployed configurations from local SQLite. Each Instance Actor spawns its child Script and Alarm Actors.
  - Data collection (Data Connection Layer re-establishes subscriptions as Instance Actors register their data source references).
  - Store-and-forward delivery (buffer is already replicated locally).
- Active debug view streams from central are interrupted — the engineer must re-open them.
- Health reporting resumes from the new active node.
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).

## Downing Strategy (auto-down — availability-first)

**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:

- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues (as before). If the **active/oldest** node crashes, the younger survivor becomes leader among the reachable members, downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — **no operator action and no victim restart required**. This closes the keep-oldest total-outage gap.
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.

### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)

- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
- **static-quorum (quorum-size 1)**: Akka's `IsTooManyMembers` guard (`members > quorum*2-1`, i.e. `2 > 1`) returns **DownAll** on any unreachability — total shutdown, strictly worse.
- **keep-majority**: a 1-vs-1 split keeps the side with the lowest address, which just moves the fatal crash from "the oldest" to "the lowest-address node".
- **lease-majority**: needs a shared lease store (Kubernetes API, SQL, …) reachable from both nodes — not available at sites (one node per VM, no shared infrastructure).

### Downed-node recovery

When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:

1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
4. The restarted process rejoins as a **fresh incarnation**, and can re-form the cluster on its own if no peer is reachable — see Seed Node Ordering below.

### Seed Node Ordering

**Every node lists ITSELF as `seed-nodes[0]` and its partner second (decision 2026-07-22).**

Akka runs two different bootstrap processes depending on that first entry. When `seed-nodes[0]` is the node's own address it runs `FirstSeedNodeProcess`: it `InitJoin`s the *other* seeds and self-joins only after `seed-node-timeout` elapses with nobody answering. When it is not, the node runs `JoinSeedNodeProcess`, which can never form a new cluster — it retries `InitJoin` indefinitely.

Until 2026-07-22 every node listed the same first seed, so a node that had to **boot alone** — a cold start of only the non-first-seed VM, or the survivor crashing while its peer was still dead — never reached `Up` and was never routable. That was the **registered outage gap**, and recovery was operator-driven. Self-first ordering closes it using Akka's own protocol, and `StartupValidator` fails the boot if a node config ever breaks the ordering (the invariant is silent when violated, so it is enforced loudly).

Behavior, covered by `SelfFirstSeedBootstrapTests` (real in-process clusters built from `BuildHocon` at production failure-detection timings):

| Scenario | Behavior |
|---|---|
| Lone cold-start, peer dead | Self-joins after `seed-node-timeout` (~5s) — operational, unattended |
| Restart into a **live** peer | The peer answers `InitJoinAck`; the node joins the existing cluster and never islands |
| Both cold-start simultaneously, mutually reachable | The `InitJoin` handshake resolves it *before* either self-joins → **one** 2-member cluster |
| Both cold-start during a genuine boot-time **partition** | Each forms its own cluster — the same dual-active class `auto-down` already accepts, same recovery (restart one side) |

**Rejected alternative — an external self-form timer.** A watchdog that waited a configurable window for membership and then called `Cluster.Join(SelfAddress)` was implemented and discarded. Its success signal ("am I `Up` yet?") cannot distinguish *no seed answered* from *a seed answered and the join is in flight*, because it sits outside Akka's join handshake. On a routine standby restart the peer is alive but the join stalls behind removal of the restarting node's own stale incarnation (the peer must down it, then wait for the failure detector and a leader action); a `Join(self)` issued during `TryingToJoin` abandons the in-flight join and forms a second cluster at the same address. Measured: a permanent split that had not healed after 90s, converting a routine restart into an outage of the previously-healthy node. Akka's first-seed process has no such race because it is part of the handshake, which is why the ordering — not a timer — is the mechanism.

The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.

### Simultaneous cold start — the bootstrap guard (opt-in, Gitea #33)

Self-first-on-both has a residual cost. When **both** nodes cold-start at the same instant, each is `seed-nodes[0]` for itself, so **each runs `FirstSeedNodeProcess`**, times out waiting for the other, and forms its **own** single-node cluster — a split brain (two oldest members, two singletons, dual-active) that does **not** auto-merge and persists until an operator restarts one side. In-process loopback tests usually converge (the `InitJoin` handshake resolves before either self-join deadline, so row 3 above passes), which is exactly why the split hides until a real deployment powers up both VMs truly in parallel — a shared power event or a hypervisor host reboot. The sister project **OtOpcUa** (same `ZB.MOM.WW.*` Akka topology) reproduced it reliably on its docker rig.

The **bootstrap guard** eliminates the split without giving up cold-start-alone. It is an **opt-in dark switch** — `ScadaBridge:Cluster:BootstrapGuard:Enabled`, **default off**, so a node with the guard disabled keeps Akka's config-driven self-first auto-join byte-identical. When enabled, the node starts with **no config seed nodes** (`AkkaHostedService.BuildHocon` emits an empty seed list, so Akka does not auto-join) and a coordinator (`ClusterBootstrapCoordinator`, an `IHostedService` registered in both the Central and Site composition roots) picks the join order after a reachability probe:

- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins self-first and forms immediately if no peer answers — no probe, no delay. The tie-break is **case-insensitive** (a hostname-casing mismatch between the two configs must never make both think they are the founder and re-open the split).
- The **higher** node TCP-probes its partner's Akka port (a node binds its port well before it forms) up to `PartnerProbeSeconds`: **reachable ⇒ peer-first** (join the founder, never race it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so cold-start-alone is preserved for the higher node too).
- The order is decided **before** a single `Cluster.JoinSeedNodes`, from an explicit reachability signal. The coordinator **never re-forms mid-handshake** — that was the failure mode of the rejected self-form timer above (a `Join(self)` on a bare timeout is not ignored mid-handshake; it wins and islands a node).

**Residual trade-off (accepted, operator-visible).** Once the higher node commits peer-first it cannot self-form (`JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window, the higher node hangs unjoined; the coordinator logs a WARNING after a bounded grace, and a **restart** recovers it (the guard re-runs, finds the founder down, and forms alone). The guard deliberately does not re-decide mid-handshake.

| Key | Default | Meaning |
|---|---|---|
| `BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere (single-node install, self absent, 3+ seeds). |
| `BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot when the guard is on. |
| `BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. Validated `> 0` when enabled. |
| `BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. Validated `> 0` when enabled. |

Decision core (`ClusterBootstrapGuard`) is a pure, fully unit-tested function; the probe + `JoinSeedNodes` runtime (`ClusterBootstrapCoordinator`) is covered by real-ActorSystem tests including the load-bearing higher-node-cold-start-alone case and the headline both-cold-start-together-form-one-cluster case (`ClusterBootstrapCoordinatorTests`). The alternative is purely operational (stagger the two VMs' service-manager start / start the founder first); compose `depends_on` was assumed to do that on the docker rig, but it demonstrably does not — see below. Ported from OtOpcUa (`lmxopcua` commit `d1dac87f`); implementing it in both products keeps their failure/recovery model identical, since a shared power event hits both pairs at once.

**Live status (2026-08-02): ENABLED on the primary docker rig (all 8 nodes), live gate PASS.** With the guard off, `docker/deploy.sh`'s simultaneous container recreate reproduced the split naturally TWICE on 2026-08-01 — first run split both site-b and site-c (mutual `InitJoinNack`, each node "JOINING itself and forming a new cluster"), second run split site-c while the other pairs converged: the race is a per-pair coin flip, and compose `depends_on` does not serialize far enough to prevent it. With the guard enabled on every rig node, two consecutive simultaneous-start trials (a full redeploy plus a full-topology `docker compose restart`) converged all four pairs deterministically — every lower-address node logged founder/self-first, every higher node logged partner-REACHABLE/peer-first and received `Welcome from` the founder; zero splits, `/health/ready` green. This is the issue-acceptance live gate that was deferred at implementation time. The switch stays default-off in the Host appsettings and everywhere else (wonder-app-vd03, docker-env2).

### Manual Failover (admin-triggered)

An Administrator can swap the central pair's roles deliberately — for planned maintenance on the active node, or to move singletons off a node behaving badly without waiting for a crash. The control is the **Trigger failover** button on the central-cluster card of the Health dashboard (`/monitoring/health`).

Semantics:

- **Graceful `Leave`, never `Down`.** The active node leaves the cluster so `ClusterSingletonManager` hands its singletons to the standby before the member is removed. A `Down` would skip the hand-off.
- **Target = the oldest Up member with the `Central` role**, the same rule `ActiveNodeEvaluator` uses, so the node acted on is exactly the one hosting the singletons — never Akka's cluster *leader*, whose address-ordered definition diverges from singleton placement after a restart.
- **Peer guard.** Refused when fewer than two Up `Central` members exist: failing over a lone node is an outage, not a failover. Enforced server-side in `AkkaManualFailoverService`; the button is also disabled client-side when the pair has no online standby.
- **Admin-only** (`AuthorizationPolicies.RequireAdmin`). The Health page itself is all-roles, so the gate lives on the control.
- **Audited before acting.** One `AuditChannel.Cluster` / `AuditKind.ManualFailover` row is written through `ICentralAuditWriter` *before* the Leave is issued, naming the actor and the target address — the acting node can be the one that goes away, and an audit written afterwards could be lost to the shutdown it describes. Audit failure never blocks the failover.
- **Your own page will disconnect.** Traefik routes the UI to the active node, so triggering a failover drops the admin's Blazor circuit; it reconnects against the new active node. The confirmation dialog says so.

After the Leave the node's `ActorSystem` terminates, the `WhenTerminated` watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member (the new standby). Recovery is the normal restart contract above — no interaction with seed ordering, since the peer is alive on this path.

#### Site-pair failover

The same control appears on each **site** card. Central and each site are separate Akka clusters, so central cannot act on a site's membership — it *asks*, over the central→site gRPC command channel (`SiteCommandService`):

1. `CommunicationService.TriggerSiteFailoverAsync` sends a `TriggerSiteFailover` inside a `SiteEnvelope`.
2. The site's `SiteCommunicationActor` (registered per node, so contact rotation reaches whichever answers) resolves the target from cluster state and issues the graceful `Leave` locally.
3. It replies `SiteFailoverAck` — sent **before** the Leave takes effect, so the ack still arrives when the acking node is the one leaving.

Differences from central failover, all deliberate:

- **Role scope is `site-{SiteId}`, not the base `Site` role.** Site singletons (the Deployment Manager) are placed on the site-specific role; using the base role would move the wrong node. Pinned by both a unit test asserting the role string and a real-cluster test.
- **Misroute is refused.** A command whose `SiteId` does not match the receiving node's is rejected rather than acted on — a misroute must never fail over a site the operator did not select.
- **A fault is acked, not thrown.** Reporting through the ack keeps the reason; letting it reach supervision would restart the communication actor and reduce central's Ask to a bare timeout.
- **The operator's own session is unaffected** — a different cluster entirely. The confirmation dialog therefore does *not* carry the "this page will disconnect" warning that the central one does.
- **Refusal vs unreachable are distinct.** A `false` ack is a definitive answer from the site (peer guard, misroute); a timeout means the site never answered. Both surface to the operator with their own wording, because only one of them means "nothing happened".

**Rolling upgrade.** A site running a binary older than this contract has no handler for `TriggerSiteFailover`; the message dead-letters and central's Ask times out, reported as "site did not respond". That is the correct user-facing outcome — an old site genuinely cannot honour the request. Message evolution stays additive-only.

## Single-Node Operation

`akka.cluster.min-nr-of-members` must be set to **1**. After failover, only one node is running. If set to 2, the surviving node waits for a second member before allowing the Cluster Singleton (Site Runtime Deployment Manager) to start — blocking all data collection and script execution indefinitely.

## Failure Detection Timing

Configurable defaults for heartbeat and failure detection:

| Setting | Default | Description |
|---------|---------|-------------|
| Heartbeat interval | 2 seconds | Frequency of health check messages between nodes |
| Failure detection threshold | 10 seconds | Time without heartbeat before a node is considered unreachable |
| Stable-after (split-brain) | 15 seconds | Time cluster must be stable before resolver acts |
| **Total failover time** | **~25 seconds** | Detection (10s) + stable-after (15s) + singleton restart |

These values balance failover speed with stability — fast enough that data collection gaps are small, tolerant enough that brief network hiccups don't trigger unnecessary failovers.

## Dual-Node Recovery

If both nodes in a cluster fail simultaneously (e.g., site power outage):

1. **No manual intervention required.** Since both nodes are configured as seed nodes, whichever node starts first forms a new cluster (self-first ordering) and the second node joins when it starts. If both power up truly simultaneously, each self-first seed can run `FirstSeedNodeProcess` and form its own 1-node cluster — a split brain; enable the opt-in **bootstrap guard** (Gitea #33, see *Simultaneous cold start* above) on co-located VM pairs that share a power domain to make them converge on one cluster instead.
2. **State recovery** (each node has its own local copy of all required data):
   - **Site clusters**: The Deployment Manager singleton reads deployed configurations from local SQLite and re-creates the full Instance Actor hierarchy. Store-and-forward buffers are already persisted locally. Alarm states re-evaluate from incoming data values.
   - **Central cluster**: All state is in MS SQL (configuration database). The active node resumes normal operation.
3. The keep-oldest resolver handles the "both starting fresh" case naturally — there is no pre-existing cluster to conflict with.

## Graceful Shutdown & Singleton Handover

When a node is stopped for planned maintenance (Windows Service stop), `CoordinatedShutdown` triggers a **graceful leave** from the cluster. This enables the Cluster Singleton (Site Runtime Deployment Manager) to hand over to the other node in seconds (limited by the hand-over retry interval) rather than waiting for the full failure detection timeout (~25 seconds).

Configuration required:
- `akka.coordinated-shutdown.run-by-clr-shutdown-hook = on`
- `akka.cluster.run-coordinated-shutdown-when-down = on`

The Host component wires CoordinatedShutdown into the Windows Service lifecycle (see REQ-HOST-9).

## Node Configuration

Each node is configured with:
- **Cluster seed nodes**: **Both nodes** are seed nodes — each node lists both itself and its partner. Either node can start first and form the cluster; the other joins when it starts. No startup ordering dependency.
- **Cluster role**: Central or Site (plus site identifier for site clusters).
- **Akka.NET remoting**: Hostname/port for inter-node and inter-cluster communication (default 8081 central, 8082 site).
- **gRPC port** (site nodes only): Dedicated HTTP/2 port for the SiteStreamGrpcServer (default 8083). Separate from the Akka remoting port — gRPC uses Kestrel, Akka uses its own TCP transport.
- **Local storage paths**: SQLite database locations (site nodes only).

## Windows Service

- Each node runs as a **Windows service** for automatic startup and recovery.
- Service configuration includes Akka.NET cluster settings and component-specific configuration.

## Platform

- **OS**: Windows Server.
- **Runtime**: .NET (Akka.NET).
- **Cluster**: Akka.NET Cluster (application-level, not Windows Server Failover Clustering).

## Dependencies

- **Akka.NET**: Core actor system, cluster, remoting, and cluster singleton libraries.
- **Windows**: Service hosting, networking.
- **MS SQL** (central only): Database connectivity.
- **SQLite** (sites only): Local storage.

## Interactions

- **All components**: Every component runs within the Akka.NET actor system managed by this infrastructure.
- **Site Runtime**: The Deployment Manager singleton relies on Akka.NET cluster singleton support provided by this infrastructure.
- **Communication Layer**: Built on top of the Akka.NET remoting provided here.
- **Health Monitoring**: Reports node status (active/standby) as a health metric.
