lmxopcua/docs/Redundancy.md

Redundancy (v2)

Overview

OtOpcUa supports OPC UA non-transparent warm/hot redundancy. Two or more OtOpcUa.Host processes run side-by-side, share the same Config DB, and join the same Akka.NET cluster. Each process owns a distinct ApplicationUri; OPC UA clients discover both endpoints by reading Server.ServerArray (NodeId i=2254) on either node and pick one based on the ServiceLevel byte that each server publishes.

Discovery surface. The ServerArray path on the Server object is what each node populates with self + peer ApplicationUris — see OpcUaApplicationHost.PopulateServerArray and the per-node PeerApplicationUris option below. The redundancy-object-type ServerUriArray proper (a child of Server.ServerRedundancy) remains deferred pending an SDK object-type upgrade; clients should read Server.ServerArray for peer discovery today.

v2 change. v1's operator-managed ClusterNode.RedundancyRole column + RedundancyCoordinator / ApplyLeaseRegistry / PeerHttpProbeLoop are gone. Primary/secondary is now derived from Akka cluster membership — the oldest Up member carrying the driver role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.

The runtime pieces live in:

Component	Project	Role
RedundancyStateActor	OtOpcUa.ControlPlane.Redundancy	Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts RedundancyStateChanged on the redundancy-state DPS topic.
OpcUaPublishActor	OtOpcUa.Runtime.OpcUa	Per-driver-node; subscribes to the redundancy-state topic, computes a health-aware ServiceLevel byte via ServiceLevelCalculator (see below), and forwards it to IServiceLevelPublisher.
IServiceLevelPublisher / SdkServiceLevelPublisher	OtOpcUa.Commons.OpcUa / OtOpcUa.OpcUaServer	Writes the byte into the SDK's Server.ServiceLevel Variable. Production binds DeferredServiceLevelPublisher, which swaps in the real SdkServiceLevelPublisher once the SDK is up (it needs IServerInternal, available only after StandardServer.Start); until then writes route through NullServiceLevelPublisher.
ServiceLevelCalculator	OtOpcUa.Cluster.Redundancy (Core.Cluster)	Pure function (NodeHealthInputs) → byte — the DB/probe-aware tiering (see truth table below). Covered by ServiceLevelCalculatorTests. Now the live publish path — OpcUaPublishActor calls it on every HealthTick and RedundancyStateChanged event. Moved to Core.Cluster so Runtime can reach it without a Runtime→ControlPlane reference.
DbHealthProbeActor	OtOpcUa.Runtime.Health	Per-node; runs SELECT 1 against ConfigDb every 5s. Read by the health endpoint AND by OpcUaPublishActor (the DbReachable ServiceLevel input).
PeerProbeSupervisor	OtOpcUa.Runtime.Health	Per-node; subscribes to the redundancy-state topic and maintains one PeerOpcUaProbeActor child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers.
PeerOpcUaProbeActor	OtOpcUa.Runtime.Health	Spawned by PeerProbeSupervisor; pings a peer opc.tcp://peer:4840 with a TCP connect (2s timeout) and publishes OpcUaProbeResult on the redundancy-state topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe.
ClusterRoleInfo	OtOpcUa.Cluster	Live view of cluster membership + role-leader; exposes IClusterRoleInfo to the rest of the host. (Akka's role-leader is not what elects the redundancy Primary — see below.)

ServiceLevel tiers

Health-aware tiering (ServiceLevelCalculator — live path)

ServiceLevelCalculator.Compute(NodeHealthInputs) is the live publish path. OpcUaPublishActor calls it on every HealthTick (~5 s) and on each RedundancyStateChanged snapshot, then forwards the result through IServiceLevelPublisher to the SDK's Server.ServiceLevel Variable.

The four inputs are sourced locally per driver node:

Input	Source
MemberState	Local SelfMember.Status from the Akka cluster (Up / Joining / Leaving / …).
DbReachable	Local DbHealthProbeActor — OpcUaPublishActor Asks it on each HealthTick; an Ask timeout is treated as Reachable=false.
OpcUaProbeOk	Result of a peer probing THIS node's OPC UA endpoint: PeerProbeSupervisor spawns one PeerOpcUaProbeActor per OTHER driver-role peer; each probe publishes OpcUaProbeResult(probed-node, ok) on the redundancy-state topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → true (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT false demotes.
Stale (derived)	!DbReachable || (now − lastDbHealth.AsOfUtc) > 30 s || (now − snapshotEntry.AsOfUtc) > 30 s.
IsDriverPrimary	The local node's entry in the RedundancyStateChanged snapshot from RedundancyStateActor.

The resulting truth table (all tiers are now reachable at runtime):

Tier	Byte	Condition
Down / Detached	0	Member status is not Up or Joining (leaving, removed, exiting), OR node has no driver role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255.
Critically degraded	100	ConfigDb unreachable AND data is stale.
Stale	200	Data stale but ConfigDb reachable.
Healthy follower	240	DB reachable + OPC UA probe ok + not stale + not the driver Primary.
Healthy primary	250	Same as healthy follower + this node is the driver Primary (+10 bonus).

Secondary 100 → 240 (behavior change). Previously a healthy Secondary published 100 (coarse role-only mapping). It now publishes 240 — both nodes sit at 240/250 under healthy conditions, with the leader still preferred by the +10 bonus. Clients with the standard "pick highest ServiceLevel" heuristic continue to prefer the primary.

Backward-compatible fallback (legacy seam)

A node with no DbHealthStatus wired (e.g. early bootstrap window before the first DbHealthProbeActor reply) falls back to the old role-only mapping: Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the first DbHealthStatus arrives the calculator takes over. The first computed ServiceLevel (even 0) is always published so no node lingers at the SDK default 255.

Roles come from RedundancyStateActor.BuildSnapshot: a node with the driver role is Primary when it is the oldest Up member carrying the driver role, otherwise Secondary; a node without the driver role is Detached.

Oldest, not role leader (changed 2026-07-21). The election previously used ClusterState.RoleLeader("driver") — the lowest-addressed Up member with the role. Akka offers both, and they name different nodes: role leader is ordered by address (host, then port), oldest by up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node re-joins as the youngest while keeping its address, so if it holds the lower address it becomes role leader. Meanwhile ClusterSingletonManager places singletons on the oldest, so the old derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by RedundancyPrimaryElectionTests, which forms a real two-node cluster where the oldest node holds the higher address so the two orderings genuinely disagree. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary to stop the name asserting the old derivation.

KNOWN LIMITATION — the election is per Akka cluster, not per application Cluster. The election yields exactly one Primary across the whole Akka cluster. A fleet that runs several application clusters (each with its own redundant pair) inside one Akka cluster — which is what docker-dev/docker-compose.yml models, with MAIN, SITE-A and SITE-B — therefore has one Primary among all its driver nodes, not one per pair. Every consumer of the role inherits this: ServiceLevel, the alerts emit gate, and the inbound device-write gate below.

The unit of redundancy everywhere else in the product is the application Cluster (ClusterId) — it owns drivers, devices and ClusterNode rows, and the LocalDb deployment-artifact cache is already keyed by it so that a pair shares one entry. Scoping the election the same way is the fix; RedundancyStateActor is an admin-role singleton in the same assembly as AdminOperationsActor, which already reads and groups ClusterNodes by cluster.

Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node in the fleet — see docs/plans/2026-07-20-localdb-phase2-live-gate.md. The alarm drain no longer depends on this (it defers only to a node that shares its queue), but the other three gates still do.

Data flow

Cluster topology event ──────────────────────────────────────────┐
                                                                   ▼
                                               RedundancyStateActor (admin singleton)
                                                                   │  debounce 250ms
                                                                   ▼
                                               DPS topic "redundancy-state"
                                                    │                         ▲
                            ┌───────────────────────┘                         │
                            │                                                  │
                            ▼                                                  │
              Driver node: OpcUaPublishActor                                   │
              ┌─────────────────────────────────────────────────────────┐      │
              │  Inputs collected per ~5s HealthTick:                   │      │
              │   • MemberState  ← Akka SelfMember.Status               │      │
              │   • DbReachable  ← DbHealthProbeActor (Ask, timeout→F) │      │
              │   • OpcUaProbeOk ← OpcUaProbeResult about THIS node    │──────┘
              │   • Stale        ← derived from above timestamps        │  PeerProbeSupervisor
              │   • IsLeader     ← RedundancyStateChanged snapshot      │  → PeerOpcUaProbeActor(s)
              │                                                         │  publish OpcUaProbeResult
              │  ServiceLevelCalculator.Compute(NodeHealthInputs)       │  on "redundancy-state"
              │  → byte (0/100/200/240/250)                             │
              └───────────────────────────────────────────────────────-─┘
                            │
                            ▼
              IServiceLevelPublisher (SdkServiceLevelPublisher)
                            │
                            ▼
              OPC UA Server.ServiceLevel Variable

Both DbHealthProbeActor and PeerOpcUaProbeActor feed the live publish path. The peer probe publishes OpcUaProbeResult on the redundancy-state topic; OpcUaPublishActor consumes only results whose target is itself and applies freshness-debouncing before passing them to the calculator. DbHealthProbeActor is queried directly via Ask on each HealthTick.

The admin singleton is the cluster's only RedundancyStateActor. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current Cluster.State. There is no DB-persisted state to recover.

Configuration

Per-node identity comes from appsettings.json + the OTOPCUA_ROLES env var:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
    "SeedNodes": [
      "akka.tcp://otopcua@node-a.lan:4053",
      "akka.tcp://otopcua@node-b.lan:4053"
    ],
    "Roles": ["admin", "driver"]
  }
}

OTOPCUA_ROLES=admin,driver

Both nodes share the same ConfigDb connection string; Cluster.PublicHostname + Roles are what makes them distinct in cluster gossip. List both peers in SeedNodes on both nodes so either can cold-start alone — and list each node ITSELF first: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See Bootstrap: self-first seed ordering.

There is no longer a Node:NodeId setting and no ClusterNode.RedundancyRole column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as host:port of the cluster PublicHostname (see ClusterRoleInfo.LocalNode for the formula).

RedundancyStateActor NodeId consistency (fixed). RedundancyStateActor now keys each node's NodeRedundancyState entry by the canonical host:port node id (via a ToNodeId(Address) helper mirroring ClusterRoleInfo.ToNodeId). Previously it keyed by member.Address.Host (host-only, e.g. central-2); since every subscriber matches by the canonical host:port form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes RedundancyStateActor consistent with the stated contract above. Additionally, RedundancyStateActor now re-publishes the current snapshot on a periodic heartbeat (default 10 s) so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).

The ClusterNode.ServiceLevelBase column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by ServiceLevelCalculator from cluster role and live health inputs, independent of this stored preference.

Peer URI advertising

Each node advertises its partner via OpcUaApplicationHostOptions.PeerApplicationUris (an IList<string>, default empty). OpcUaApplicationHost.PopulateServerArray appends each configured peer URI to the SDK's IServerInternal.ServerUris string table after server startup, so that Server.ServerArray reads served by OnReadServerArray return both self + peers. The options bind from the OpcUa config section (see Program.cs — AddValidatedOptions<OpcUaApplicationHostOptions>(…, "OpcUa")). Set this per-node in appsettings.json:

{
  "OpcUa": {
    "PeerApplicationUris": ["urn:node-b:OtOpcUa"]
  }
}

Node A lists Node B's ApplicationUri and vice-versa. Validated by DualEndpointTests in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ — boots two OpcUaApplicationHost instances on loopback, asserts a real OPCFoundation client Session reading Server.ServerArray from Node A sees both URIs.

Split-brain / downing

The default downing strategy is auto-down (changed 2026-07-21). It is selected by Cluster:SplitBrainResolverStrategy, and ServiceCollectionExtensions.BuildDowningHocon turns that into the akka.cluster.downing-provider-class actually installed:

Cluster:SplitBrainResolverStrategy	Provider installed	Posture
auto-down (default)	Akka.Cluster.AutoDowning with auto-down-unreachable-after = 15s	Availability. The leader among the reachable members downs the unreachable peer, so a crash of either node — oldest included — fails over in place.
keep-oldest	Akka.Cluster.SBR.SplitBrainResolverProvider reading the split-brain-resolver block in akka.conf	Partition-safety. A partition can never run dual-active. Not safe for a 2-node pair — see below.

Any other value fails the host at startup rather than falling through to a default.

Why keep-oldest is no longer the default

The previous documentation in this section — and the code comments behind it — asserted that keep-oldest with down-if-alone = on was "the correct strategy for a 2-node warm-redundancy pair." That was wrong, and the error was not conservative: it described a total-outage configuration as the recommended one.

In Akka.NET 1.5.62's KeepOldest.OldestDecision, the down-if-alone rescue branch requires the surviving side to hold ≥ 2 members. In a two-node cluster a lost peer is always a 1-vs-1 split, so the branch never fires and the lone survivor falls through to DownReachable — downing itself — and run-coordinated-shutdown-when-down = on then terminates it. down-if-alone is a 3+-node feature; it does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members. See ../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md for the upstream decision record, which OtOpcUa follows.

The alternatives do not help a pair: static-quorum with quorum 1 hits Akka's IsTooManyMembers guard and returns DownAll; with quorum 2 the survivor downs itself; keep-majority keeps the lowest-address side, which just moves the fatal crash from "the oldest node" to "the other node"; lease-majority needs a shared lease store both nodes can reach.

The accepted trade

Under auto-down, a genuine network partition (both nodes alive, link cut) leaves each side downing the other and continuing alone: both run active, both hold singletons, and both advertise the primary ServiceLevel. Recovery is operator-driven — after the link heals, restart one side; it rejoins as a fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual downing quarantines the association.

This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and a crash under keep-oldest was unrecoverable without operator action anyway.

Deployments that would rather take an outage than ever run dual-active can set Cluster:SplitBrainResolverStrategy: "keep-oldest" — but should read the section above first, because for a two-node pair that choice means any crash of the oldest node is a full outage.

Recovery still depends on supervision

1. A service supervisor that restarts the process on exit — production Install-Services.ps1 sets sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000; the docker-dev rig sets restart: unless-stopped.
2. The recovery watchdog ActorSystemTerminationWatchdog (registered in Program.cs after AddAkka) — it watches ActorSystem.WhenTerminated and, on an unexpected self-down, calls IHostApplicationLifetime.StopApplication() so the process exits and the supervisor restarts it instead of idling forever with a dead actor system. Under auto-down the survivor no longer needs this (it is never the one downed); the downed node still does.
3. Both peers listed in SeedNodes on every node, each node listing itself first, so a restarted node can re-join via either peer and a node cold-starting while its peer is dead still forms the cluster. Without the self-first ordering it waits in InitJoin forever (auto-down removes the crash outage, not that one) — see Bootstrap: self-first seed ordering.

On HardKillFailoverTests: that in-process test hard-kills the oldest and asserts the survivor becomes sole driver role-leader. It passed even under keep-oldest — because it simulates the crash with provider.Transport.Shutdown(), which leaves node A's ActorSystem alive (a transport partition with the oldest still running), not a real process death. A real 2-container docker kill of the oldest downed the survivor (verified 2026-07-15, archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md). It is a standing example of a green test over a fatal defect: keep the distinction between a transport partition and a process death in mind when reading it.

SplitBrainResolverActivationTests pins the strategy by starting a real host through the production bootstrap and reading akka.cluster.downing-provider-class back off the running ActorSystem — asserting the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the losing one still looks correct in the file.

Failover during a partition is what the cluster and supervisor do automatically. For a planned swap — patching, draining a node — see Manual failover below.

Manual failover

The cluster redundancy page (/clusters/{id}/redundancy) carries a Live redundancy panel — the current driver Primary and every Up driver member, read from live cluster state — and an admin-gated Trigger failover button.

The mechanism is a graceful cluster Leave of the current Primary, never a Down: IManualFailoverService.FailOverDriverPrimaryAsync selects the oldest Up driver member — the identical query to RedundancyStateActor.SelectDriverPrimary, pinned together by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary so the node acted on is exactly the node the election names. The leaving node hands its singletons over through the cluster-leave phases, CoordinatedShutdown runs, ActorSystemTerminationWatchdog exits the process, the supervisor restarts it, and it rejoins as the youngest member — so the survivor is now the oldest, becomes Primary, and advertises ServiceLevel 250. OPC UA clients re-select.

Rule	Behaviour
Authorization	AdminUiPolicies.FleetAdmin (Administrator only) — gated in markup and re-checked server-side before the call. Deliberately not ConfigEditor, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer.
Peer guard	Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it.
Confirmation	A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select).
Audit	Written through the shared ZB.MOM.WW.Audit.IAuditWriter seam before the Leave — action cluster.manual-failover, actor, target. A refused failover changes nothing and writes nothing.

Mesh-scope caveat. Until the per-cluster mesh work lands (docs/plans/2026-07-21-per-cluster-mesh-design.md), the Primary is elected once per Akka mesh rather than per application Cluster row — so on a fleet running several clusters in one mesh this button acts on the whole mesh's Primary, which may belong to a different cluster than the page you are on. The panel says so. Phase 6 of the mesh program removes both the caveat and this notice.

Live gate outstanding. The auto-down change is verified by unit tests against the effective configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has not yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster; assert the survivor stays up, takes the singletons, and reports the primary ServiceLevel) should run alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See docs/plans/2026-07-21-per-cluster-mesh-design.md §6.2 / Phase 0a.

Bootstrap: self-first seed ordering

Listing both peers in SeedNodes does not mean either node can cold-start alone — and neither does any downing strategy: auto-down removes the crash outage, not this one. Akka runs a different bootstrap process depending on whether seed-nodes[0] is the node's own address:

seed-nodes[0] is…	Process Akka runs	Can it form a NEW cluster?
this node's own address	FirstSeedNodeProcess	Yes — it InitJoins the other seeds and self-joins once seed-node-timeout (5 s) passes with nobody answering.
some other node	JoinSeedNodeProcess	No — it retries InitJoin forever, however long the peer stays down.

So the ordering is the mechanism. Every node that is a seed of its own mesh lists ITSELF first and its partner second (decision 2026-07-22):

// central-1                                     // central-2
"SeedNodes": [                                   "SeedNodes": [
  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
]                                                ]

The rule is conditional: it binds only when a node's own address appears in its own seed list. A driver-only site node is seeded solely by central-1 (today's docker-dev topology) — it is legitimately not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every site node in the fleet.

AkkaClusterOptionsValidator enforces it at startup (AddValidatedOptions → ValidateOnStart in AddOtOpcUaCluster), because the invariant otherwise fails silently: the process runs, the port listens, the node simply never becomes a member. Identity is compared on Cluster:PublicHostname (falling back to Cluster:Hostname when blank) and Cluster:Port — the address Akka puts in SelfAddress. Comparing against Cluster:Hostname would be worse than wrong: in docker-dev it is 0.0.0.0, which matches no seed URI anywhere, so the rule would silently exempt the entire rig.

Sequential recovery is island-free by construction: a FirstSeedNodeProcess node self-joins only when no seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both nodes cold-starting simultaneously converge on one cluster while they are mutually reachable — the InitJoin handshake resolves it before either self-join deadline expires. The residual risk is a boot-time partition (both cold-starting, mutually unreachable): both form, which is the same dual-active class the auto-down strategy already accepts, with the same recovery (restart one side).

Pinned by SelfFirstSeedBootstrapTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/), which starts real in-process clusters through the production bootstrap at production failure-detection timings: a lone cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting together converge on one cluster. Its falsifiability control — the old peer-first ordering, same scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control self-join proving the node was formable all along. AkkaClusterOptionsValidatorTests covers the validator, including the site-node exemption and the 0.0.0.0 identity trap.

Why the self-form watchdog was retired (2026-07-22)

The first fix for this gap was Cluster:SelfFormAfter + ClusterBootstrapFallback: wait 10 s for membership, then call Cluster.Join(SelfAddress). Both are deleted. A timer sits outside Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" — and Cluster.Join(SelfAddress) is not ignored mid-handshake: it wins.

Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence). The watchdog islanded a node that manual failover had bounced. central-1 restarted, received InitJoinAck from central-2 at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window, because the peer's ring still held its previous incarnation (Exiting → Down → Removed, retired at 10:49:06). At 10:49:15 the watchdog fired and the node formed a second cluster, islanded until an operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of that plan collided. A TCP reachability guard (ea45ace1) patched that one shape — never self-form while a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely passed: central-2 cold-started alone in 10 s and a non-seed site-a-2 correctly stayed out.

The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because the decision is made inside the handshake, by Akka, on the actual InitJoin replies. It was also inert for every non-seed node, so after this change it could never usefully fire. SelfFormBootstrapTests is replaced by SelfFirstSeedBootstrapTests, whose Restarting_node_rejoins_its_live_peer_instead_of_islanding is the drilled scenario above.

Live gate outstanding for the new mechanism. Self-first ordering is verified here by in-process clusters at production timings, and on the sister project (ScadaBridge 4a6341d8) on real clusters and live docker containers. It has not been re-drilled on the OtOpcUa docker-dev rig since the swap — the central-2 cold-start-alone drill should be repeated on the rebuilt image. Owned by docs/plans/2026-07-22-per-cluster-mesh-program.md Phase 7 alongside the auto-down 1-vs-1 gate.

Primary data-plane gate (writes, acks, alerts emit)

Three data-plane surfaces are Primary-gated so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator writes (DriverHostActor.HandleRouteNodeWrite), native-alarm acks (HandleRouteNativeAlarmAck), and the native/scripted alerts emit (ForwardNativeAlarm / ScriptedAlarmHostActor.OnEngineEmission). All three route through one policy — PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount) (OtOpcUa.Runtime.Drivers):

Local role (last redundancy snapshot)	Cluster driver members	Decision
Primary	any	service
Secondary / Detached	any	deny
unknown (no snapshot yet, or snapshot omitted this node)	≤ 1 (single-node / boot on a lone driver)	service (boot-window / single-node posture)
unknown	≥ 2 (a real driver peer exists)	deny (default-deny — a Primary peer exists; don't act until a snapshot proves this node is it)

Before archreview 03/S4 the unknown-role case defaulted to allow unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s RedundancyStateActor heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a multi-driver cluster now default-denies. The driver member count is read live from Cluster.State.Members (Up members with the driver role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), DriverHostActor logs a one-time Warning so the silent mismatch is diagnosable.

What a client sees during the boot window (multi-node, role unknown): an inbound write is rejected (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 AuditWriteUpdateEvent; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window ("not primary (role unknown)") from a steady-state Secondary ("not primary"). Acks are dropped (Warning-logged when role-unknown). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.

Under warm/hot redundancy both cluster nodes run ScriptedAlarmHostActor and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on /alerts and duplicate historian writes, only the Primary node publishes externally:

• alerts topic emission — ScriptedAlarmHostActor and DriverHostActor.ForwardNativeAlarm subscribe to the redundancy-state DPS topic and cache the local node's RedundancyRole, then gate the cluster alerts publish through PrimaryGatePolicy (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain ungated on both nodes so the secondary is always ready to serve clients after a failover.
• HistorianAdapterActor historization — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the alerts DPS topic and translates each AlarmTransitionEvent → AlarmHistorianEvent before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
• The alarm sink's DRAIN — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. LocalDbStoreAndForwardSink takes a drainGate fed from IRedundancyRoleView — a singleton DriverHostActor publishes its PrimaryGatePolicy verdict to, because the drain runs on a timer and cannot receive RedundancyStateChanged itself. A gated-off node reports HistorianDrainState.NotPrimary. Delivery is at-least-once across a failover by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See AlarmHistorian.md.

Net effect: each alarm transition appears once on /alerts and would historize once, not once per node.

See ScriptedAlarms.md and AlarmTracking.md for the scripted-alarm engine internals.

Pair-local store (LocalDb — Phases 1 + 2)

Independently of the ServiceLevel machinery above, every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database. Phase 2 added the alarm store-and-forward buffer (alarm_sf_events) to it alongside the config cache, so a node that dies holding undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above and AlarmHistorian.md. Phase 1 caches the deployed-configuration artifact (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver node restarting into a central-SQL-Server outage would come up with no configuration at all. With the cache, it boots from the last artifact it applied instead, and logs a running-from-cache signal.

• What replicates. The cache can optionally replicate to the node's redundant pair peer over the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate: deployment_artifacts and deployment_pointer. Replication is default-OFF and fail-closed — it stays inert until LocalDb:SyncListenPort + a peer are configured, and a missing/mismatched LocalDb:Replication:ApiKey refuses or silently halts sync. It is enabled per-pair as an opt-in.
• The replicated-cache payoff. In a pair with replication on, either node can be the one that applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a central outage can boot the current config even if it never applied that deploy itself — its peer did, and the row replicated.
• What it does NOT cover. The cache is a boot fallback for central-SQL outages only, not a standby data path and not a replacement for central. A new deployment still requires central SQL; the cache is read only when the central fetch fails at startup, and the node resumes fresh-config behavior once central returns. It does not replicate live tag values, alarms, or historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate logic: a node running from cache still participates in redundancy normally.

Full detail: the enablement runbook at docs/operations/2026-07-20-localdb-pair-replication.md and the LocalDb section of docs/Configuration.md.

Client-side failover

The OtOpcUa Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI supports -F / --failover-urls for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See Client.CLI.md.

Observability

OpcUaPublishActor emits one metric on every ServiceLevel transition (it suppresses no-op repeats of the same byte):

Metric	Type	Notes
otopcua.redundancy.service_level_change	Counter ({change})	OPC UA Server.ServiceLevel transitions emitted by the redundancy state. Tagged with level = the new byte.
otopcua.redundancy.primary_gate_denied	Counter ({denial})	Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged site = write | ack | alarm-emit and reason = secondary | detached | role-unknown. A non-zero role-unknown rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering).
otopcua.opcua.apply.failed	Counter ({apply})	Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged kind = rebuild (the rebuild threw) | nodes (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — OpcUaPublishActor also logs the degraded apply at Error.
galaxy.writes.advise_failed	Counter ({write})	Galaxy writes short-circuited to Bad because AdviseSupervisory failed — the value could not have committed, so the write fails closed (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter ZB.MOM.WW.OtOpcUa.Driver.Galaxy.
galaxy.writes.unconfirmed	Counter ({write})	Galaxy writes reported Good off an empty gateway statuses array (command accepted; the COM-side commit is unconfirmed pending the mxaccessgw WriteComplete correlation follow-up). The honest signal for the optimistic-Good rate until that cross-repo fix lands. On the Galaxy driver meter.

The redundancy/apply meters are defined on OtOpcUaTelemetry (src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs); the Galaxy write meters hang off the driver's own meter (EventPump.MeterName). All surface through whatever OpenTelemetry exporter the host configures.

Galaxy fail-closed write semantics (archreview 06/S-1): a non-secured Galaxy write commits only when its item is AdviseSupervisory-advised first (the writer runs with no login). If the advise fails, the write is not issued — GatewayGalaxyDataWriter returns BadCommunicationError (a knowingly-lost write is never reported as Good). An empty statuses array on the write reply is still treated as Good but metered as galaxy.writes.unconfirmed. See Historian.md / the Galaxy driver notes for the write path.

Depth reference

For the full design — message contracts, tiered calculator truth table, recovery semantics — see docs/plans/2026-05-26-akka-hosting-alignment-design.md §6.