lmxopcua/docs/Redundancy.md

# Redundancy (v2)

## Overview

OtOpcUa supports OPC UA **non-transparent** warm/hot redundancy. Two or more `OtOpcUa.Host` processes run side-by-side, share the same Config DB, and join the same Akka.NET cluster. Each process owns a distinct `ApplicationUri`; OPC UA clients discover both endpoints by reading `Server.ServerArray` (NodeId `i=2254`) on either node and pick one based on the `ServiceLevel` byte that each server publishes.

> **Discovery surface.** The `ServerArray` path on the `Server` object is what each node populates with self + peer `ApplicationUri`s — see `OpcUaApplicationHost.PopulateServerArray` and the per-node `PeerApplicationUris` option below. The redundancy-object-type `ServerUriArray` proper (a child of `Server.ServerRedundancy`) remains deferred pending an SDK object-type upgrade; clients should read `Server.ServerArray` for peer discovery today.

> **v2 change.** v1's operator-managed `ClusterNode.RedundancyRole` column + `RedundancyCoordinator` / `ApplyLeaseRegistry` / `PeerHttpProbeLoop` are gone. Primary/secondary is now derived from **Akka cluster membership** — the oldest Up member carrying the `driver` role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.

The runtime pieces live in:

| Component | Project | Role |
|---|---|---|
| `RedundancyStateActor` | `OtOpcUa.ControlPlane.Redundancy` | Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts `RedundancyStateChanged` on the `redundancy-state` DPS topic. |
| `OpcUaPublishActor` | `OtOpcUa.Runtime.OpcUa` | Per-driver-node; subscribes to the `redundancy-state` topic, computes a health-aware ServiceLevel byte via `ServiceLevelCalculator` (see below), and forwards it to `IServiceLevelPublisher`. |
| `IServiceLevelPublisher` / `SdkServiceLevelPublisher` | `OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer` | Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`. |
| `ServiceLevelCalculator` | `OtOpcUa.Cluster.Redundancy` (`Core.Cluster`) | Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. **Now the live publish path** — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference. |
| `DbHealthProbeActor` | `OtOpcUa.Runtime.Health` | Runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input). **Spawned only when a ConfigDb is present** (admin-role or fused admin+driver nodes) — a driver-only node has no `IDbContextFactory<OtOpcUaConfigDbContext>` to probe, so this actor is not spawned there at all; see "DB-less nodes" below. |
| `PeerProbeSupervisor` | `OtOpcUa.Runtime.Health` | Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers. |
| `PeerOpcUaProbeActor` | `OtOpcUa.Runtime.Health` | Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe. |
| `ClusterRoleInfo` | `OtOpcUa.Cluster` | Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is *not* what elects the redundancy Primary — see below.) |

## ServiceLevel tiers

### Health-aware tiering (`ServiceLevelCalculator` — live path)

`ServiceLevelCalculator.Compute(NodeHealthInputs)` is the live publish path.
`OpcUaPublishActor` calls it on every `HealthTick` (~5 s) and on each
`RedundancyStateChanged` snapshot, then forwards the result through
`IServiceLevelPublisher` to the SDK's `Server.ServiceLevel` Variable.

The four inputs are sourced locally per driver node:

| Input | Source |
|---|---|
| `MemberState` | Local `SelfMember.Status` from the Akka cluster (Up / Joining / Leaving / …). |
| `DbReachable` | Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`. |
| `OpcUaProbeOk` | Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes. |
| `Stale` (derived) | `!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`. |
| `IsDriverPrimary` | The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`. |

The resulting truth table (all tiers are now reachable at runtime):

| Tier | Byte | Condition |
|---|---|---|
| Down / Detached | 0 | Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255. |
| Critically degraded | 100 | ConfigDb unreachable AND data is stale. |
| Stale | 200 | Data stale but ConfigDb reachable. |
| Healthy follower | 240 | DB reachable + OPC UA probe ok + not stale + not the driver Primary. |
| Healthy primary | 250 | Same as healthy follower + this node is the driver Primary (+10 bonus). |

> **Secondary 100 → 240 (behavior change).** Previously a healthy Secondary
> published 100 (coarse role-only mapping). It now publishes **240** — both
> nodes sit at 240/250 under healthy conditions, with the leader still preferred
> by the +10 bonus. Clients with the standard "pick highest ServiceLevel"
> heuristic continue to prefer the primary.

#### DB-less (driver-only) nodes — per-cluster mesh Phase 4

**Client-visible change.** Per-cluster mesh Phase 4 cut the ConfigDb connection off driver-only
nodes (Akka role `driver`, not `admin`) — `Program.cs` registers `AddOtOpcUaConfigDb` only when
`hasAdmin`. With no ConfigDb to probe, `DbHealthProbeActor` is **not spawned** on such a node
(`ServiceCollectionExtensions.WithOtOpcUaRuntimeActors` skips it whenever the resolved
`IDbContextFactory<OtOpcUaConfigDbContext>` is null), and `OpcUaPublishActor` runs in a `dbLess`
mode instead of the health-aware path above:

- `DbReachable` is fed **constant `true`** — LocalDb is the config store and is in-process, so it
  can never be "unreachable" the way a remote SQL Server can.
- `Stale` is derived from the `redundancy-state` snapshot age alone (`(now − snapshotEntry.AsOfUtc)
  > 30 s`) — there is no DB-health term to fold in.
- `OpcUaProbeOk` and `IsDriverPrimary` are unchanged (peer OPC UA probe, oldest-Up-member election).

**Consequence:** a healthy driver-only node publishes **240** as a follower or **250** as the
driver Primary **even when central SQL is unreachable** — the survive-alone posture the fetch-and-
cache design (Phase 3) exists to support. Contrast this with a `Direct`/DB-backed node (fused
admin+driver, or any node still reading ConfigDb directly): an unreachable ConfigDb there still
drives `DbReachable=false`, which forces `Stale=true` in the derivation above and the node drops to
**100** (or **0** once the sample itself goes stale) via the health-aware tiering, not 240/250.
Same byte, opposite meaning depending on which kind of node is publishing it — a client or operator
reading ServiceLevel in isolation cannot tell "central is down and I don't care" (driver-only, 240)
from "central is down and I'm degraded" (DB-backed, 100/0) without also knowing the node's role
shape. `ServiceLevelCalculator.Compute` itself is unchanged; only the inputs a DB-less node feeds
it changed — see `OpcUaPublishActor.RecomputeServiceLevel`'s `_dbLess` branch.

#### Backward-compatible fallback (legacy seam)

A node with no `DbHealthStatus` wired (e.g. early bootstrap window before the
first `DbHealthProbeActor` reply) falls back to the old role-only mapping:
Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the
first `DbHealthStatus` arrives the calculator takes over. The first computed
ServiceLevel (even 0) is always published so no node lingers at the SDK default
255.

Roles come from `RedundancyStateActor.BuildSnapshot`: a node with the `driver`
role is `Primary` when it is the **oldest Up member carrying the `driver` role**, otherwise
`Secondary`; a node without the `driver` role is `Detached`.

**Oldest, not role leader (changed 2026-07-21).** The election previously used
`ClusterState.RoleLeader("driver")` — the *lowest-addressed* Up member with the role. Akka offers
both, and they name different nodes: role leader is ordered by address (host, then port), oldest by
up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node
re-joins as the *youngest* while keeping its address, so if it holds the lower address it becomes
role leader. Meanwhile `ClusterSingletonManager` places singletons on the **oldest**, so the old
derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data
plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by
`RedundancyPrimaryElectionTests`, which forms a real two-node cluster where the oldest node holds the
*higher* address so the two orderings genuinely disagree. `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` to stop the name asserting the old derivation.

**Accepted residual (#620): the election's role match is case-sensitive.** `RedundancyStateActor`
delegates to `ZB.MOM.WW.Health.Akka`'s `ClusterActiveNode.OldestUpMember`, which matches roles with
Akka's own ordinal `Member.HasRole`. A peer gossiping a miscased plain role (`Driver`) is therefore
invisible to the election — which only happens in a mixed rolling-upgrade window with a pre-R3-04
build (fixed builds normalize roles before gossip). Deliberately **not** patched: the library is a
family-shared package, and making either it or a local role pre-filter case-insensitive would split the
Primary election from `ClusterSingletonManager` placement (also ordinal) — recreating the "Primary not
on the singleton node" defect above. Case-sensitive on both sides is the consistent behaviour; finish
the rolling upgrade rather than working around it.

> **RESOLVED by Phase 6 — the election is now per application `Cluster`, not per Akka mesh.**
> Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent
> 2-node mesh per application `Cluster` (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)
> below). `RedundancyStateActor` is now a `cluster-{ClusterId}`-scoped singleton spawned on every driver
> node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe
> no longer exists. See `docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md`.
>
> Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the
> alarm-history drain on every node in the fleet — see `docs/plans/2026-07-20-localdb-phase2-live-gate.md`.

## Per-cluster meshes (Phase 6)

The fleet no longer runs one Akka mesh. Phase 6 split it into **three independent 2-node meshes** — the
central pair (roles `admin,driver,cluster-MAIN`), the site-a pair (`driver,cluster-SITE-A`), and the
site-b pair (`driver,cluster-SITE-B`) — all sharing the same `ActorSystem` name (`otopcua`) on one network,
separated purely by **per-pair self-first seed partitioning**: each node's `Cluster:SeedNodes` lists only
itself and its own pair partner, so gossip never crosses a pair boundary (see
[Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering)).

Consequences:

- **One Primary per pair — two or more Primaries fleet-wide, by design.** Each mesh runs its own
  election, so a three-mesh fleet has three Primaries, one per application `Cluster`. This is correct,
  not a regression of the old single-Primary assumption.
- **`RedundancyStateActor` is a `cluster-{ClusterId}`-scoped singleton**, spawned on every driver node
  (falling back to the plain `driver` role for a legacy node not carrying a cluster role). Each pair
  elects and publishes its own `redundancy-state` on its own mesh's DPS — the DPS topic name is
  unchanged, but it no longer reaches past the pair.
- **`ClusterNodeAddressReconciler` is own-cluster-scoped** — an admin node can only reconcile
  `ClusterNode` rows belonging to its own cluster; it has no gossip visibility into another mesh to
  reconcile against.
- **`CentralCommunicationActor` dials each node directly over gRPC** — one `NodeCommandService.Execute`
  call per enabled, non-maintenance `ClusterNode` row carrying a `GrpcPort` — rather than fanning
  through any client/receptionist object (the Akka `ClusterClient` transport this used to run on is
  **deleted**; see [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md)).
  Central reaches a site mesh over gRPC for commands (`MeshTransport`), telemetry (see
  `docs/Telemetry.md`), and config fetch (`ConfigServe` — see
  [`ConfigSource`/`ConfigServe`](../CLAUDE.md#config-source-configsource--configserve)), never over
  gossip.
- **`SplitTopologyTransportValidator` fails host start** on a node carrying a `cluster-{ClusterId}`
  role unless it is also configured for the split-safe transports: `MeshTransport:Mode = Grpc`,
  `Telemetry:Mode = Grpc` (driver nodes), `TelemetryDial:Mode = Grpc` (admin nodes). A DPS-mode transport
  on a cluster-role node would silently try to gossip across a partition that no longer exists, so the
  validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
- **The compiled transport-mode defaults were deliberately NOT flipped** — `MeshTransport:Mode`,
  `Telemetry:Mode`, and `TelemetryDial:Mode` all still default to `Dps`. A blast-radius assessment
  found flipping the compiled default would break every integration test and every existing
  `appsettings.json` profile that doesn't explicitly set these keys. The guardrail is the validator
  above plus explicit per-deployment configuration, not a default flip.

### Secrets replication is pair-local

Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its
own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by
design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a
cross-mesh sync through).

## Data flow

```
Cluster topology event ──────────────────────────────────────────┐
                                                                   ▼
                                               RedundancyStateActor (admin singleton)
                                                                   │  debounce 250ms
                                                                   ▼
                                               DPS topic "redundancy-state"
                                                    │                         ▲
                            ┌───────────────────────┘                         │
                            │                                                  │
                            ▼                                                  │
              Driver node: OpcUaPublishActor                                   │
              ┌─────────────────────────────────────────────────────────┐      │
              │  Inputs collected per ~5s HealthTick:                   │      │
              │   • MemberState  ← Akka SelfMember.Status               │      │
              │   • DbReachable  ← DbHealthProbeActor (Ask, timeout→F) │      │
              │   • OpcUaProbeOk ← OpcUaProbeResult about THIS node    │──────┘
              │   • Stale        ← derived from above timestamps        │  PeerProbeSupervisor
              │   • IsLeader     ← RedundancyStateChanged snapshot      │  → PeerOpcUaProbeActor(s)
              │                                                         │  publish OpcUaProbeResult
              │  ServiceLevelCalculator.Compute(NodeHealthInputs)       │  on "redundancy-state"
              │  → byte (0/100/200/240/250)                             │
              └───────────────────────────────────────────────────────-─┘
                            │
                            ▼
              IServiceLevelPublisher (SdkServiceLevelPublisher)
                            │
                            ▼
              OPC UA Server.ServiceLevel Variable
```

Both `DbHealthProbeActor` and `PeerOpcUaProbeActor` feed the live publish path.
The peer probe publishes `OpcUaProbeResult` on the `redundancy-state` topic;
`OpcUaPublishActor` consumes only results whose target is itself and applies
freshness-debouncing before passing them to the calculator. `DbHealthProbeActor`
is queried directly via Ask on each `HealthTick`.

The admin singleton is the cluster's only `RedundancyStateActor`. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current `Cluster.State`. There is no DB-persisted state to recover.

## Configuration

Per-node identity comes from `appsettings.json` + one role source — `Cluster:Roles` (below) or the
`OTOPCUA_ROLES` env var; either works, and if BOTH are set on a node they must agree (see
[`Configuration.md`](Configuration.md) → `EffectiveRoles`, archreview R3-04):

```jsonc
{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
    "SeedNodes": [
      "akka.tcp://otopcua@node-a.lan:4053",
      "akka.tcp://otopcua@node-b.lan:4053"
    ],
    "Roles": ["admin", "driver"]
  }
}
```

the `Cluster__Roles__0`/`Cluster__Roles__1` env-var equivalent, or (the fallback, only when
`Cluster:Roles` is absent/empty):

```
OTOPCUA_ROLES=admin,driver
```

Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).

There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).

> **`RedundancyStateActor` NodeId consistency (fixed).** `RedundancyStateActor` now keys each node's `NodeRedundancyState` entry by the canonical `host:port` node id (via a `ToNodeId(Address)` helper mirroring `ClusterRoleInfo.ToNodeId`). Previously it keyed by `member.Address.Host` (host-only, e.g. `central-2`); since every subscriber matches by the canonical `host:port` form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes `RedundancyStateActor` consistent with the stated contract above. Additionally, `RedundancyStateActor` now **re-publishes the current snapshot on a periodic heartbeat (default 10 s)** so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).

The `ClusterNode.ServiceLevelBase` column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by `ServiceLevelCalculator` from cluster role and live health inputs, independent of this stored preference.

### Peer URI advertising

Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicationUris` (an `IList<string>`, default empty). `OpcUaApplicationHost.PopulateServerArray` appends each configured peer URI to the SDK's `IServerInternal.ServerUris` string table after server startup, so that `Server.ServerArray` reads served by `OnReadServerArray` return both self + peers. The options bind from the `OpcUa` config section (see `Program.cs` — `AddValidatedOptions<OpcUaApplicationHostOptions>(…, "OpcUa")`). Set this per-node in `appsettings.json`:

```json
{
  "OpcUa": {
    "PeerApplicationUris": ["urn:node-b:OtOpcUa"]
  }
}
```

Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.

## Split-brain / downing

**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
into the `akka.cluster.downing-provider-class` actually installed:

| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
|---|---|---|
| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |

Any other value **fails the host at startup** rather than falling through to a default.

### Why keep-oldest is no longer the default

The previous documentation in this section — and the code comments behind it — asserted that
`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
**That was wrong, and the error was not conservative: it described a total-outage configuration as the
recommended one.**

In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
for the upstream decision record, which OtOpcUa follows.

The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
a shared lease store both nodes can reach.

### The accepted trade

Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
downing quarantines the association.

This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
a crash under `keep-oldest` was unrecoverable without operator action anyway.

Deployments that would rather take an outage than ever run dual-active can set
`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
two-node pair that choice means *any* crash of the oldest node is a full outage.

### Recovery still depends on supervision

1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
   `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
   `restart: unless-stopped`.
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
   it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
   `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
   of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
   (it is never the one downed); the **downed** node still does.
3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
   node can re-join via either peer and a node cold-starting while its peer is dead still forms the
   cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
   outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).

> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
> partition and a process death in mind when reading it.

`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
losing one still looks correct in the file.

Failover during a partition is what the cluster and supervisor do automatically. For a *planned* swap —
patching, draining a node — see [Manual failover](#manual-failover) below.

## Manual failover

The cluster redundancy page (`/clusters/{id}/redundancy`) carries a **Live redundancy** panel — the current
driver Primary and every Up `driver` member, read from live cluster state — and an admin-gated **Trigger
failover** button.

The mechanism is a graceful cluster **`Leave`** of the current Primary, never a `Down`:
`IManualFailoverService.FailOverDriverPrimaryAsync` selects the **oldest Up `driver` member** — the identical
query to `RedundancyStateActor.SelectDriverPrimary`, pinned together by
`ManualFailoverServiceTests.Parity_with_SelectDriverPrimary` so the node acted on is exactly the node the
election names. The leaving node hands its singletons over through the cluster-leave phases,
`CoordinatedShutdown` runs, `ActorSystemTerminationWatchdog` exits the process, the supervisor restarts it,
and it rejoins as the **youngest** member — so the survivor is now the oldest, becomes Primary, and
advertises `ServiceLevel` 250. OPC UA clients re-select.

| Rule | Behaviour |
|---|---|
| Authorization | `AdminUiPolicies.FleetAdmin` (**Administrator only**) — gated in markup *and* re-checked server-side before the call. Deliberately not `ConfigEditor`, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer. |
| Peer guard | Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it. |
| Confirmation | A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select). |
| Audit | Written through the shared `ZB.MOM.WW.Audit.IAuditWriter` seam **before** the Leave — action `cluster.manual-failover`, actor, target. A *refused* failover changes nothing and writes nothing. |

Phase 6 of the mesh program (see [Per-cluster meshes (Phase 6)](#per-cluster-meshes-phase-6)) split
the fleet into one 2-node mesh per application `Cluster`, so the button now acts on **this pair's own**
Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.

> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It
> was **not** drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single
> six-node mesh where the 1-vs-1 pathology cannot occur. **Phase 6 has landed and every mesh is now
> exactly two nodes**, so the drill (kill the oldest container of a genuine two-node cluster; assert
> the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) is finally
> testable — deferred to Phase 7's failover drill. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.

## Bootstrap: self-first seed ordering

Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
process depending on whether `seed-nodes[0]` is the node's own address:

| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
|---|---|---|
| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |

So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
partner second** (decision 2026-07-22):

```jsonc
// central-1                                     // central-2
"SeedNodes": [                                   "SeedNodes": [
  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
]                                                ]
```

The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
site node in the fleet.

`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
no seed URI anywhere, so the rule would silently exempt the entire rig.

Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member.

**Simultaneous cold-start CAN split — this is the residual gap the bootstrap guard below closes.** When
BOTH nodes start from nothing at the same instant, each runs `FirstSeedNodeProcess`, and if neither has
formed before the other's `seed-node-timeout` expires, EACH self-joins and forms its own single-node
cluster — two Primaries in one pair. On in-process loopback the `InitJoin` handshake usually resolves fast
enough to converge, but the Phase 6 live gate reproduced the split reliably on the docker-dev rig (real
container start timing + DNS): both nodes logged `JOINING itself … forming a new cluster` and both advertised
ServiceLevel 250. Two independent clusters do **not** auto-merge, so the split persists until an operator
restarts one side.

### Bootstrap guard (`Cluster:BootstrapGuard`, opt-in)

The guard eliminates the simultaneous-start split without giving up cold-start-alone. It is a **dark switch,
default off** — a node with `Cluster:BootstrapGuard:Enabled=false` (the default) keeps Akka's config-driven
self-first auto-join exactly as above.

When enabled, the node is given **no config seed nodes** (so Akka does not auto-join), and
`ClusterBootstrapCoordinator` picks the join order from a deterministic address tie-break plus a reachability
probe:

- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins
  self-first and forms immediately if no peer answers — no probe, no delay.
- The **higher** node probes its partner's Akka port (a plain TCP connect; the partner binds its port well
  before it forms) for up to `PartnerProbeSeconds` (default 25 s): **reachable ⇒ peer-first** (it joins the
  founder, never races it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so
  it forms alone — cold-start-alone preserved for the higher node too).

The order is decided **before** issuing a single `Cluster.JoinSeedNodes`, from an explicit reachability
signal — it never re-decides mid-handshake, the failure mode that retired the `SelfFormAfter` watchdog.

**Residual trade-off (accepted, operator-visible):** once the higher node commits peer-first it cannot
self-form (Akka's `JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window,
the higher node hangs unjoined; the coordinator logs a clear WARNING after a bounded grace, and a **restart
recovers it** (the guard re-runs, finds the founder down, and forms alone). Config keys:

| Key | Default | Notes |
|---|---|---|
| `Cluster:BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere. |
| `Cluster:BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot. |
| `Cluster:BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. |
| `Cluster:BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. |

On the docker-dev rig the **site-a pair is the enablement demo** (guard on, startup serialization removed so
both nodes cold-start simultaneously); **site-b keeps the `depends_on: service_healthy` startup serialization
with the guard off**, as the A/B control. Either mechanism prevents the split; the guard is the one that also
works on production hardware, where compose `depends_on` does not exist.

The alternative to the guard is **operational**: stagger the two VMs' service-manager start, or bring the
designated founder VM up first — see the Phase 7 co-located operator runbook
(`docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).

The residual risk that remains even with the guard is a boot-time **partition** (both cold-starting, mutually
unreachable from the start): both form, which is the same dual-active class the `auto-down` strategy already
accepts, with the same recovery (restart one side).

Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
in-process clusters through the production bootstrap at production failure-detection timings: a lone
cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
including the site-node exemption and the `0.0.0.0` identity trap.

### Why the self-form watchdog was retired (2026-07-22)

The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.

> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.

The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
is the drilled scenario above.

> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.

## Primary data-plane gate (writes, acks, alerts emit)

Three data-plane surfaces are **Primary-gated** so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator **writes** (`DriverHostActor.HandleRouteNodeWrite`), native-alarm **acks** (`HandleRouteNativeAlarmAck`), and the native/scripted **alerts emit** (`ForwardNativeAlarm` / `ScriptedAlarmHostActor.OnEngineEmission`). All three route through one policy — `PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount)` (`OtOpcUa.Runtime.Drivers`):

| Local role (last redundancy snapshot) | Cluster driver members | Decision |
|---|---|---|
| `Primary` | any | **service** |
| `Secondary` / `Detached` | any | **deny** |
| unknown (no snapshot yet, or snapshot omitted this node) | ≤ 1 (single-node / boot on a lone driver) | **service** (boot-window / single-node posture) |
| unknown | ≥ 2 (a real driver peer exists) | **deny** (default-deny — a Primary peer exists; don't act until a snapshot proves this node is it) |

Before archreview 03/S4 the unknown-role case defaulted to *allow* unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s `RedundancyStateActor` heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a **multi-driver** cluster now default-**denies**. The driver member count is read live from `Cluster.State.Members` (Up members with the `driver` role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), `DriverHostActor` logs a one-time Warning so the silent mismatch is diagnosable.

**What a client sees during the boot window (multi-node, role unknown):** an inbound write is **rejected** (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 `AuditWriteUpdateEvent`; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window (`"not primary (role unknown)"`) from a steady-state Secondary (`"not primary"`). Acks are dropped (Warning-logged when role-unknown) — and since review R1 m2 a native OPC UA Acknowledge is refused up front (`BadInvalidState`) in the same window: `DriverHostActor` publishes its Primary verdict to `IDriverPrimaryRoleSource` (the native-ack gate, and the MQTT Sparkplug primary-host claim) once at start and again on every cluster membership change, not only on redundancy snapshots, so the published verdict denies as soon as a driver peer is Up rather than sitting on the single-node "allow" seed until the first snapshot. ⚠️ "As soon as a driver peer is Up" means **in this node's own cluster view**, i.e. after it has joined: the host is spawned before the join, when the view holds no members, and a member count of 0 takes the single-node *allow* posture — for the published verdict and the live gate alike. So a node cold-starting into a pair whose peer is already Primary allows until its own join completes (seconds normally; up to `Cluster:BootstrapGuard:PartnerProbeSeconds`, 25 s, under the bootstrap guard), and in that window the MQTT Sparkplug primary-host claim can be held on both nodes. Native ack is not exposed (the view and the route agree, so there is no acked → unacked revert). Denying until self is Up was not adopted: it would also deny writes/acks during a single-node cluster's own join (review R6 MINOR-2). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.

Under warm/hot redundancy both cluster nodes run `ScriptedAlarmHostActor` and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on `/alerts` and duplicate historian writes, only the Primary node publishes externally:

- **`alerts` topic emission** — `ScriptedAlarmHostActor` and `DriverHostActor.ForwardNativeAlarm` subscribe to the `redundancy-state` DPS topic and cache the local node's `RedundancyRole`, then gate the cluster `alerts` publish through `PrimaryGatePolicy` (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain **ungated** on both nodes so the secondary is always ready to serve clients after a failover.
- **`HistorianAdapterActor` historization** — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the `alerts` DPS topic and translates each `AlarmTransitionEvent` → `AlarmHistorianEvent` before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
- **The alarm sink's DRAIN** — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. `LocalDbStoreAndForwardSink` takes a `drainGate` fed from `IRedundancyRoleView` — a singleton `DriverHostActor` publishes its `PrimaryGatePolicy` verdict to, because the drain runs on a timer and cannot receive `RedundancyStateChanged` itself. A gated-off node reports `HistorianDrainState.NotPrimary`. The gate applies only to a node whose queue a peer actually holds — `LocalDbReplicationPolicy.SharesRowsWithPeer`: `LocalDb:Replication:ApiKey` AND (`PeerAddress` OR `LocalDb:SyncListenPort` > 0); any other node always drains (review R1 m3). Within a pair BOTH halves stand down for their partner: the dialling half knows it by the `PeerAddress` host, the passive half (no `PeerAddress`) as its per-cluster mesh partner (review R6 MINOR-4 — before that the passive half never stood down, so both halves drained and every event reached the historian twice; a passive half with no `cluster-{ClusterId}` role still cannot identify its partner and keeps draining). Delivery is **at-least-once across a failover** by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See [AlarmHistorian.md](AlarmHistorian.md).

- **Continuous-historization value CAPTURE** (Gitea #537) — a third gate on the same role rule, and the
  one that gates the *opposite* half of its pipeline. Both pair nodes serve the same address space and see
  the same dependency-mux fan-out, so an ungated `ContinuousHistorizationRecorder` wrote every live sample
  twice. It now reads a capture gate per value, fed from `PrimaryGatePolicy.ShouldRecordHistorizedValues`
  via `IHistorizationRoleView` (a third verdict on the same `RedundancyRoleView` object, registered
  unconditionally on driver nodes rather than alongside the alarm section's `IRedundancyRoleView`).
  Gating the **capture** rather than the drain is deliberate: unlike the alarm queue, the historization
  outbox is a node-local FasterLog that replicates to nobody, so a drain-only gate would leave the standby
  appending forever to a queue no one empties. Values a node already captured are still drained by that
  node. ⚠️ **The peer it stands down for must record the same refs** — a Primary in its own per-cluster
  mesh — not merely hold the same rows; a node with no cluster-scoped role never stands down, because a
  fleet-wide election's Primary usually runs a different cluster's configuration and deferring to it would
  end this cluster's historization silently. See [Historian.md](Historian.md).

Net effect: each alarm transition appears **once** on `/alerts` and would historize once, not once per node.

See [ScriptedAlarms.md](ScriptedAlarms.md) and [AlarmTracking.md](AlarmTracking.md) for the scripted-alarm engine internals.

## Command transport (central↔node)

Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node
subscribes, and the two ends must be members of the same cluster. Per-cluster mesh **Phase 2**
(cross-mesh cutover: [ClusterClient→gRPC migration](plans/2026-08-06-clusterclient-to-grpc-design.md))
adds a second transport under `MeshTransport:Mode` (see
[Configuration.md § `MeshTransport`](Configuration.md#meshtransport-centralnode-command-transport))
that does not need shared membership — the prerequisite for splitting the fleet into one mesh per
application `Cluster`. The Akka `ClusterClient` transport that originally filled this second slot is
**deleted**; `Grpc` is its replacement, not an alternative alongside it.

| | `Dps` (default, pair-local) | `Grpc` (cross-mesh) |
|---|---|---|
| Central → node commands | `Publish` on `deployments` / `driver-control` / `alarm-commands` | `NodeCommandService.Execute`, one unary call per node dialled directly from `ClusterNode.Host`+`GrpcPort`; decoded and handed to the node's `NodeCommunicationActor`, which re-emits it on the node's **local** `EventStream` exactly as the DPS path does |
| Node → central acks | `Publish` on `deployment-acks` | `CentralCommandService.ReportApply`, walking `MeshTransport:CentralGrpcEndpoints` in order and stopping at the first success, forwarded to the deploy-coordinator singleton |
| Requires shared cluster membership | **yes** | no |
| Auth | none (mesh membership is the boundary) | per-cluster PSK bearer, both directions, fail-closed |

Two properties are load-bearing and easy to break:

- **The gRPC dialer fans node by node, not through any shared client or receptionist object.** There
  is no rotation and no "whichever node answers" — each command is dialled at the specific node's own
  endpoint, so a node left out of the `ClusterNode` dispatch set (no `GrpcPort`, disabled, or in
  `MaintenanceMode`) silently keeps its old configuration while the deployment still seals green
  around it, the same failure shape `ClusterClient`'s `SendToAll` had.
- **Inbound commands go to the node-local `EventStream`, never back onto a DPS topic.** DPS is
  mesh-wide and central fans out to *every* node, so a DPS re-publish would deliver N copies of
  every command to every subscriber.
- **Both transports are fire-and-forget, with no retry or queue.** A failed gRPC dispatch is logged
  (naming the node and the gRPC status) and dropped in the dispatch second — the deploy path's
  2-minute apply deadline and a node's boot-time replay (`DriverHostActor.Bootstrap()` re-applying an
  orphan `Applying` row) are the only recovery mechanisms, unchanged from the DPS/`ClusterClient` era.

Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout
are untouched, and `redundancy-state` itself stays on DPS (it is bidirectional and built from
`Cluster.State`, which makes it genuinely mesh-bound — the hardest remaining dependency before the
meshes can split).

**Observability telemetry took the opposite path (Phase 5).** The four live-panel channels
(`alerts`, `script-logs`, `driver-health`, `driver-resilience-status`) moved off DPS onto a
dedicated gRPC server-streaming contract — each driver node hosts, central dials in — selected by
`Telemetry:Mode`/`TelemetryDial:Mode` (default `Dps`). `redundancy-state` is deliberately **not**
part of that migration: it stays on DPS in every mode because it is pair-local and Cluster.State-built,
not a fleet-wide observability broadcast. See [`docs/Telemetry.md`](Telemetry.md) for the full
architecture and [`Configuration.md` § `Telemetry`/`TelemetryDial`](Configuration.md#telemetry--telemetrydial-live-telemetry-transport)
for the config keys.

**Config bytes travel out-of-band (Phase 3).** Commands stay on one of the two transports above, but
the *configuration artifact itself* does not ride any Akka message — `DispatchDeployment` is
payload-free (`DeploymentId` + `RevisionHash`). Under `ConfigSource:Mode = FetchAndCache` (see
[Configuration.md § `ConfigSource`](Configuration.md#configsource--configserve-config-fetch-and-cache))
a driver node fetches the bytes from central over a dedicated h2c **gRPC stream**, verifies
`SHA-256 == RevisionHash`, and reads only its LocalDb cache — no central-SQL config read. **Both pair
nodes fetch independently**, exactly as both read SQL today: there is no Primary gating of the fetch.
That is safe because `StoreAsync` is idempotent and the cache replicates (pair replication carries the
artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy
timing. Central serves the bytes from its SQL row; a `FetchAndCache` node that cannot reach central
fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots
its served state from the LocalDb pointer with no central dependency at all.

## Pair-local store (LocalDb — Phases 1 + 2)

Independently of the ServiceLevel machinery above, every **driver-role** node keeps a consolidated
[`ZB.MOM.WW.LocalDb`](../CLAUDE.md) SQLite database. Phase 2 added the **alarm store-and-forward
buffer** (`alarm_sf_events`) to it alongside the config cache, so a node that dies holding
undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above
and [AlarmHistorian.md](AlarmHistorian.md). Phase 1 caches the **deployed-configuration
artifact** (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver
node restarting into a **central-SQL-Server outage** would come up with no configuration at all. With
the cache, it **boots from the last artifact it applied** instead, and logs a running-from-cache
signal. A cached artifact whose bytes are empty or malformed is **no answer** (#601): nothing is applied,
the node does **not** adopt that revision or claim running-from-cache, and it goes **Stale** (retrying the
ConfigDb) — so a later dispatch of that same revision really applies instead of being acked as already
current (review R1 m4). A `FetchAndCache` node lands Steady-with-no-revision instead, as for an empty cache.

- **What replicates.** The cache can *optionally* replicate to the node's redundant pair peer over
  the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate:
  `deployment_artifacts` and `deployment_pointer`. Replication is **default-OFF and fail-closed** —
  it stays inert until `LocalDb:SyncListenPort` + a peer are configured, and a missing/mismatched
  `LocalDb:Replication:ApiKey` refuses or silently halts sync. It is enabled per-pair as an opt-in.
- **The replicated-cache payoff.** In a pair with replication on, *either* node can be the one that
  applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a
  central outage can boot the current config **even if it never applied that deploy itself** — its
  peer did, and the row replicated.
- **What it does NOT cover.** The cache is a **boot fallback for central-SQL outages only**, not a
  standby data path and not a replacement for central. A **new deployment still requires central
  SQL**; the cache is read only when the central fetch fails at startup, and the node resumes
  fresh-config behavior once central returns. It does not replicate live tag values, alarms, or
  historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate
  logic: a node running from cache still participates in redundancy normally.

Full detail: the enablement runbook at
[`docs/operations/2026-07-20-localdb-pair-replication.md`](operations/2026-07-20-localdb-pair-replication.md)
and the `LocalDb` section of [`docs/Configuration.md`](Configuration.md).

## Client-side failover

The OtOpcUa Client CLI at `src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI` supports `-F` / `--failover-urls` for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See [`Client.CLI.md`](Client.CLI.md).

## Observability

`OpcUaPublishActor` emits one metric on every ServiceLevel transition (it suppresses no-op repeats of the same byte):

| Metric | Type | Notes |
|---|---|---|
| `otopcua.redundancy.service_level_change` | Counter (`{change}`) | OPC UA `Server.ServiceLevel` transitions emitted by the redundancy state. Tagged with `level` = the new byte. |
| `otopcua.redundancy.primary_gate_denied` | Counter (`{denial}`) | Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged `site` = `write` \| `ack` \| `alarm-emit` and `reason` = `secondary` \| `detached` \| `role-unknown`. A non-zero `role-unknown` rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering). |
| `otopcua.opcua.apply.failed` | Counter (`{apply}`) | Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged `kind` = `rebuild` (the rebuild threw) \| `nodes` (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — `OpcUaPublishActor` also logs the degraded apply at **Error**. |
| `galaxy.writes.advise_failed` | Counter (`{write}`) | Galaxy writes short-circuited to `Bad` because `AdviseSupervisory` failed — the value could not have committed, so the write **fails closed** (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`. |
| `galaxy.writes.unconfirmed` | Counter (`{write}`) | Galaxy writes reported `Good` off an **empty** gateway statuses array (worker-side acceptance only — the COM commit is unconfirmed). Against a correlated gateway (mxaccessgw server ≥ `b948e69`, worker ≥ `53f69cd` — every unary write reply carries the worker's `OnWriteComplete` outcome as `statuses[0]`) this stays **flat**; an increment means the bounded completion wait (`MxGateway:Worker:WriteCompletionWaitMilliseconds`, default 1500 ms, deliberately under the 2 s Tier A Write timeout) timed out, or the gateway pre-dates the correlation. On the Galaxy driver meter. |

The redundancy/apply meters are defined on `OtOpcUaTelemetry` (`src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs`); the Galaxy write meters hang off the driver's own meter (`EventPump.MeterName`). All surface through whatever OpenTelemetry exporter the host configures.

**Galaxy fail-closed write semantics (archreview 06/S-1, closed 2026-08-09):** a non-secured Galaxy write commits only when its item is `AdviseSupervisory`-advised first (the writer runs with no login) — live-confirmed: a write through a merely plain-advised handle is refused by the automation object (`SECURITY_ERROR` detail 1008 on the correlated reply). If the advise fails, the write is **not** issued — `GatewayGalaxyDataWriter` returns `BadCommunicationError` (a knowingly-lost write is never reported as Good). With the gateway-side `OnWriteComplete` correlation live, the write reply's `statuses[0]` carries the real COM commit outcome, so a refused write surfaces as a Bad-family code and fires the #5 node revert — the writer always AddItems its **own** supervisory-advised handle (the former borrow-from-subscription-registry optimization was removed: MXAccess enforces one advise per handle, so a subscription's plain-advised handle can neither commit a no-login write nor be upgraded). An empty statuses array (completion-wait timeout / pre-correlation gateway) is still treated as `Good` but metered as `galaxy.writes.unconfirmed`. See [Historian.md](Historian.md) / the Galaxy driver notes for the write path.

## Depth reference

For the full design — message contracts, tiered calculator truth table, recovery semantics — see `docs/plans/2026-05-26-akka-hosting-alignment-design.md` §6.