lmxopcua/docs/Redundancy.md

Redundancy (v2)

Overview

OtOpcUa supports OPC UA non-transparent warm/hot redundancy. Two or more OtOpcUa.Host processes run side-by-side, share the same Config DB, and join the same Akka.NET cluster. Each process owns a distinct ApplicationUri; OPC UA clients discover both endpoints by reading Server.ServerArray (NodeId i=2254) on either node and pick one based on the ServiceLevel byte that each server publishes.

Discovery surface. The ServerArray path on the Server object is what each node populates with self + peer ApplicationUris — see OpcUaApplicationHost.PopulateServerArray and the per-node PeerApplicationUris option below. The redundancy-object-type ServerUriArray proper (a child of Server.ServerRedundancy) remains deferred pending an SDK object-type upgrade; clients should read Server.ServerArray for peer discovery today.

v2 change. v1's operator-managed ClusterNode.RedundancyRole column + RedundancyCoordinator / ApplyLeaseRegistry / PeerHttpProbeLoop are gone. Primary/secondary is now derived from Akka cluster role-leader for the driver role. The operator no longer writes a role into the DB; cluster topology (specifically the driver role-leader) drives ServiceLevel automatically.

The runtime pieces live in:

Component	Project	Role
RedundancyStateActor	OtOpcUa.ControlPlane.Redundancy	Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts RedundancyStateChanged on the redundancy-state DPS topic.
OpcUaPublishActor	OtOpcUa.Runtime.OpcUa	Per-driver-node; subscribes to the redundancy-state topic, computes a health-aware ServiceLevel byte via ServiceLevelCalculator (see below), and forwards it to IServiceLevelPublisher.
IServiceLevelPublisher / SdkServiceLevelPublisher	OtOpcUa.Commons.OpcUa / OtOpcUa.OpcUaServer	Writes the byte into the SDK's Server.ServiceLevel Variable. Production binds DeferredServiceLevelPublisher, which swaps in the real SdkServiceLevelPublisher once the SDK is up (it needs IServerInternal, available only after StandardServer.Start); until then writes route through NullServiceLevelPublisher.
ServiceLevelCalculator	OtOpcUa.Cluster.Redundancy (Core.Cluster)	Pure function (NodeHealthInputs) → byte — the DB/probe-aware tiering (see truth table below). Covered by ServiceLevelCalculatorTests. Now the live publish path — OpcUaPublishActor calls it on every HealthTick and RedundancyStateChanged event. Moved to Core.Cluster so Runtime can reach it without a Runtime→ControlPlane reference.
DbHealthProbeActor	OtOpcUa.Runtime.Health	Per-node; runs SELECT 1 against ConfigDb every 5s. Read by the health endpoint AND by OpcUaPublishActor (the DbReachable ServiceLevel input).
PeerProbeSupervisor	OtOpcUa.Runtime.Health	Per-node; subscribes to the redundancy-state topic and maintains one PeerOpcUaProbeActor child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers.
PeerOpcUaProbeActor	OtOpcUa.Runtime.Health	Spawned by PeerProbeSupervisor; pings a peer opc.tcp://peer:4840 with a TCP connect (2s timeout) and publishes OpcUaProbeResult on the redundancy-state topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe.
ClusterRoleInfo	OtOpcUa.Cluster	Live view of cluster membership + role-leader; exposes IClusterRoleInfo to the rest of the host.

ServiceLevel tiers

Health-aware tiering (ServiceLevelCalculator — live path)

ServiceLevelCalculator.Compute(NodeHealthInputs) is the live publish path. OpcUaPublishActor calls it on every HealthTick (~5 s) and on each RedundancyStateChanged snapshot, then forwards the result through IServiceLevelPublisher to the SDK's Server.ServiceLevel Variable.

The four inputs are sourced locally per driver node:

Input	Source
MemberState	Local SelfMember.Status from the Akka cluster (Up / Joining / Leaving / …).
DbReachable	Local DbHealthProbeActor — OpcUaPublishActor Asks it on each HealthTick; an Ask timeout is treated as Reachable=false.
OpcUaProbeOk	Result of a peer probing THIS node's OPC UA endpoint: PeerProbeSupervisor spawns one PeerOpcUaProbeActor per OTHER driver-role peer; each probe publishes OpcUaProbeResult(probed-node, ok) on the redundancy-state topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → true (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT false demotes.
Stale (derived)	!DbReachable || (now − lastDbHealth.AsOfUtc) > 30 s || (now − snapshotEntry.AsOfUtc) > 30 s.
IsDriverRoleLeader	The local node's entry in the RedundancyStateChanged snapshot from RedundancyStateActor.

The resulting truth table (all tiers are now reachable at runtime):

Tier	Byte	Condition
Down / Detached	0	Member status is not Up or Joining (leaving, removed, exiting), OR node has no driver role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255.
Critically degraded	100	ConfigDb unreachable AND data is stale.
Stale	200	Data stale but ConfigDb reachable.
Healthy follower	240	DB reachable + OPC UA probe ok + not stale + not role-leader.
Healthy leader	250	Same as healthy follower + this node is the driver role-leader (+10 bonus).

Secondary 100 → 240 (behavior change). Previously a healthy Secondary published 100 (coarse role-only mapping). It now publishes 240 — both nodes sit at 240/250 under healthy conditions, with the leader still preferred by the +10 bonus. Clients with the standard "pick highest ServiceLevel" heuristic continue to prefer the primary.

Backward-compatible fallback (legacy seam)

A node with no DbHealthStatus wired (e.g. early bootstrap window before the first DbHealthProbeActor reply) falls back to the old role-only mapping: Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the first DbHealthStatus arrives the calculator takes over. The first computed ServiceLevel (even 0) is always published so no node lingers at the SDK default 255.

Roles come from RedundancyStateActor.BuildSnapshot: a node with the driver role is Primary when it holds the driver role-leader lease, otherwise Secondary; a node without the driver role is Detached.

Data flow

Cluster topology event ──────────────────────────────────────────┐
                                                                   ▼
                                               RedundancyStateActor (admin singleton)
                                                                   │  debounce 250ms
                                                                   ▼
                                               DPS topic "redundancy-state"
                                                    │                         ▲
                            ┌───────────────────────┘                         │
                            │                                                  │
                            ▼                                                  │
              Driver node: OpcUaPublishActor                                   │
              ┌─────────────────────────────────────────────────────────┐      │
              │  Inputs collected per ~5s HealthTick:                   │      │
              │   • MemberState  ← Akka SelfMember.Status               │      │
              │   • DbReachable  ← DbHealthProbeActor (Ask, timeout→F) │      │
              │   • OpcUaProbeOk ← OpcUaProbeResult about THIS node    │──────┘
              │   • Stale        ← derived from above timestamps        │  PeerProbeSupervisor
              │   • IsLeader     ← RedundancyStateChanged snapshot      │  → PeerOpcUaProbeActor(s)
              │                                                         │  publish OpcUaProbeResult
              │  ServiceLevelCalculator.Compute(NodeHealthInputs)       │  on "redundancy-state"
              │  → byte (0/100/200/240/250)                             │
              └───────────────────────────────────────────────────────-─┘
                            │
                            ▼
              IServiceLevelPublisher (SdkServiceLevelPublisher)
                            │
                            ▼
              OPC UA Server.ServiceLevel Variable

Both DbHealthProbeActor and PeerOpcUaProbeActor feed the live publish path. The peer probe publishes OpcUaProbeResult on the redundancy-state topic; OpcUaPublishActor consumes only results whose target is itself and applies freshness-debouncing before passing them to the calculator. DbHealthProbeActor is queried directly via Ask on each HealthTick.

The admin singleton is the cluster's only RedundancyStateActor. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current Cluster.State. There is no DB-persisted state to recover.

Configuration

Per-node identity comes from appsettings.json + the OTOPCUA_ROLES env var:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}

OTOPCUA_ROLES=admin,driver

Both nodes share the same ConfigDb connection string; Cluster.PublicHostname + Roles are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in SeedNodes); the second node joins via the same SeedNodes list.

There is no longer a Node:NodeId setting and no ClusterNode.RedundancyRole column (the V2 migration dropped it — primary/secondary is now derived from cluster role-leadership). NodeId is derived as host:port of the cluster PublicHostname (see ClusterRoleInfo.LocalNode for the formula).

RedundancyStateActor NodeId consistency (fixed). RedundancyStateActor now keys each node's NodeRedundancyState entry by the canonical host:port node id (via a ToNodeId(Address) helper mirroring ClusterRoleInfo.ToNodeId). Previously it keyed by member.Address.Host (host-only, e.g. central-2); since every subscriber matches by the canonical host:port form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes RedundancyStateActor consistent with the stated contract above. Additionally, RedundancyStateActor now re-publishes the current snapshot on a periodic heartbeat (default 10 s) so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).

The ClusterNode.ServiceLevelBase column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by ServiceLevelCalculator from cluster role and live health inputs, independent of this stored preference.

Peer URI advertising

Each node advertises its partner via OpcUaApplicationHostOptions.PeerApplicationUris (an IList<string>, default empty). OpcUaApplicationHost.PopulateServerArray appends each configured peer URI to the SDK's IServerInternal.ServerUris string table after server startup, so that Server.ServerArray reads served by OnReadServerArray return both self + peers. The options bind from the OpcUa config section (see Program.cs — AddValidatedOptions<OpcUaApplicationHostOptions>(…, "OpcUa")). Set this per-node in appsettings.json:

{
  "OpcUa": {
    "PeerApplicationUris": ["urn:node-b:OtOpcUa"]
  }
}

Node A lists Node B's ApplicationUri and vice-versa. Validated by DualEndpointTests in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ — boots two OpcUaApplicationHost instances on loopback, asserts a real OPCFoundation client Session reading Server.ServerArray from Node A sees both URIs.

Split-brain

The split-brain resolver is active by default: Akka.Cluster.Hosting's WithClustering enables an SBR downing provider whenever ClusterOptions.SplitBrainResolver is null (it applies SplitBrainResolverOption.Default, which registers Akka.Cluster.SBR.SplitBrainResolverProvider), and that provider reads the split-brain-resolver HOCON block in akka.conf. On top of that, ServiceCollectionExtensions.BuildClusterOptions sets the typed ClusterOptions.SplitBrainResolver (KeepOldestOption { DownIfAlone = true }) to make the strategy explicit in code rather than relying on the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So the cluster is not running NoDowning; hard-crashed nodes are downed and the cluster recovers (how it recovers depends on which node was lost — see the table below). (Only an explicit NoDowning, e.g. akka.cluster.downing-provider-class = "", would leave both redundancy sides at ServiceLevel 240 indefinitely.) The HOCON block carries the tuning: active-strategy = keep-oldest, stable-after = 15s, keep-oldest.down-if-alone = on, failure-detector.threshold = 10.0 (its active-strategy + down-if-alone must stay consistent with the typed option; stable-after lives only in HOCON because the typed option can't express it).

keep-oldest is the correct strategy for a 2-node warm-redundancy pair (keep-majority/static-quorum are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover differently, and this is inherent to any 2-node cluster:

Node lost	What keep-oldest does	Recovery
Younger node (crash or partition)	The oldest is on the surviving side → it stays up and downs the younger side (DownUnreachable).	In-place, fast. The oldest keeps its singletons + driver role-leadership; RedundancyStateActor re-computes from the post-loss Cluster.State.
Oldest node (crash or partition)	The lone survivor is "the side without the oldest" → keep-oldest downs the survivor itself (DownReachable "including myself"), and run-coordinated-shutdown-when-down = on terminates it. down-if-alone does not rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature).	Exit-and-rejoin. The self-downed survivor and the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is no in-place takeover.

Recovery therefore depends on three things being in place (the ScadaBridge sister project runs the same pattern):

1. A service supervisor that restarts the process on exit — production Install-Services.ps1 sets sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000; the docker-dev rig sets restart: unless-stopped. Without it a downed node stays down and an oldest-crash looks like a total outage.
2. The recovery watchdog ActorSystemTerminationWatchdog (registered in Program.cs after AddAkka) — it watches ActorSystem.WhenTerminated and, on an unexpected self-down, calls IHostApplicationLifetime.StopApplication() so the process exits (and the supervisor restarts it) instead of idling forever with a dead actor system.
3. Both peers listed in SeedNodes on every node, so a restarted node can re-join the mesh via either peer.

On HardKillFailoverTests: that in-process test hard-kills the oldest and asserts the survivor becomes sole driver role-leader, and it passes — but it simulates the crash with provider.Transport.Shutdown(), which leaves node A's ActorSystem alive (a transport partition with the oldest still running), not a real process death. It is therefore not representative of an oldest-process crash; a real 2-container docker kill of the oldest downs the survivor (verified 2026-07-15, see archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md). Treat the recovery guarantee for the oldest as "exit-and-rejoin under supervision," not "in-place failover."

There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor do automatically. Instant in-place takeover on any single-node loss requires 3+ members (an odd cluster or a lightweight witness) — a deliberate future option, not the current 2-node posture.

Primary data-plane gate (writes, acks, alerts emit)

Three data-plane surfaces are Primary-gated so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator writes (DriverHostActor.HandleRouteNodeWrite), native-alarm acks (HandleRouteNativeAlarmAck), and the native/scripted alerts emit (ForwardNativeAlarm / ScriptedAlarmHostActor.OnEngineEmission). All three route through one policy — PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount) (OtOpcUa.Runtime.Drivers):

Local role (last redundancy snapshot)	Cluster driver members	Decision
Primary	any	service
Secondary / Detached	any	deny
unknown (no snapshot yet, or snapshot omitted this node)	≤ 1 (single-node / boot on a lone driver)	service (boot-window / single-node posture)
unknown	≥ 2 (a real driver peer exists)	deny (default-deny — a Primary peer exists; don't act until a snapshot proves this node is it)

Before archreview 03/S4 the unknown-role case defaulted to allow unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s RedundancyStateActor heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a multi-driver cluster now default-denies. The driver member count is read live from Cluster.State.Members (Up members with the driver role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), DriverHostActor logs a one-time Warning so the silent mismatch is diagnosable.

What a client sees during the boot window (multi-node, role unknown): an inbound write is rejected (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 AuditWriteUpdateEvent; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window ("not primary (role unknown)") from a steady-state Secondary ("not primary"). Acks are dropped (Warning-logged when role-unknown). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.

Under warm/hot redundancy both cluster nodes run ScriptedAlarmHostActor and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on /alerts and duplicate historian writes, only the Primary node publishes externally:

• alerts topic emission — ScriptedAlarmHostActor and DriverHostActor.ForwardNativeAlarm subscribe to the redundancy-state DPS topic and cache the local node's RedundancyRole, then gate the cluster alerts publish through PrimaryGatePolicy (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain ungated on both nodes so the secondary is always ready to serve clients after a failover.
• HistorianAdapterActor historization — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the alerts DPS topic and translates each AlarmTransitionEvent → AlarmHistorianEvent before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.

Net effect: each alarm transition appears once on /alerts and would historize once, not once per node.

See ScriptedAlarms.md and AlarmTracking.md for the scripted-alarm engine internals.

Client-side failover

The OtOpcUa Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI supports -F / --failover-urls for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See Client.CLI.md.

Observability

OpcUaPublishActor emits one metric on every ServiceLevel transition (it suppresses no-op repeats of the same byte):

Metric	Type	Notes
otopcua.redundancy.service_level_change	Counter ({change})	OPC UA Server.ServiceLevel transitions emitted by the redundancy state. Tagged with level = the new byte.
otopcua.redundancy.primary_gate_denied	Counter ({denial})	Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged site = write | ack | alarm-emit and reason = secondary | detached | role-unknown. A non-zero role-unknown rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering).
otopcua.opcua.apply.failed	Counter ({apply})	Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged kind = rebuild (the rebuild threw) | nodes (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — OpcUaPublishActor also logs the degraded apply at Error.
galaxy.writes.advise_failed	Counter ({write})	Galaxy writes short-circuited to Bad because AdviseSupervisory failed — the value could not have committed, so the write fails closed (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter ZB.MOM.WW.OtOpcUa.Driver.Galaxy.
galaxy.writes.unconfirmed	Counter ({write})	Galaxy writes reported Good off an empty gateway statuses array (command accepted; the COM-side commit is unconfirmed pending the mxaccessgw WriteComplete correlation follow-up). The honest signal for the optimistic-Good rate until that cross-repo fix lands. On the Galaxy driver meter.

The redundancy/apply meters are defined on OtOpcUaTelemetry (src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs); the Galaxy write meters hang off the driver's own meter (EventPump.MeterName). All surface through whatever OpenTelemetry exporter the host configures.

Galaxy fail-closed write semantics (archreview 06/S-1): a non-secured Galaxy write commits only when its item is AdviseSupervisory-advised first (the writer runs with no login). If the advise fails, the write is not issued — GatewayGalaxyDataWriter returns BadCommunicationError (a knowingly-lost write is never reported as Good). An empty statuses array on the write reply is still treated as Good but metered as galaxy.writes.unconfirmed. See Historian.md / the Galaxy driver notes for the write path.

Depth reference

For the full design — message contracts, tiered calculator truth table, recovery semantics — see docs/plans/2026-05-26-akka-hosting-alignment-design.md §6.