lmxopcua/docs/Redundancy.md

Redundancy (v2)

Overview

OtOpcUa supports OPC UA non-transparent warm/hot redundancy. Two or more OtOpcUa.Host processes run side-by-side, share the same Config DB, and join the same Akka.NET cluster. Each process owns a distinct ApplicationUri; OPC UA clients discover both endpoints by reading Server.ServerArray (NodeId i=2254) on either node and pick one based on the ServiceLevel byte that each server publishes.

Discovery surface. The ServerArray path on the Server object is what each node populates with self + peer ApplicationUris — see OpcUaApplicationHost.PopulateServerArray and the per-node PeerApplicationUris option below. The redundancy-object-type ServerUriArray proper (a child of Server.ServerRedundancy) remains deferred pending an SDK object-type upgrade; clients should read Server.ServerArray for peer discovery today.

v2 change. v1's operator-managed ClusterNode.RedundancyRole column + RedundancyCoordinator / ApplyLeaseRegistry / PeerHttpProbeLoop are gone. Primary/secondary is now derived from Akka cluster membership — the oldest Up member carrying the driver role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.

The runtime pieces live in:

Component	Project	Role
RedundancyStateActor	OtOpcUa.ControlPlane.Redundancy	Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts RedundancyStateChanged on the redundancy-state DPS topic.
OpcUaPublishActor	OtOpcUa.Runtime.OpcUa	Per-driver-node; subscribes to the redundancy-state topic, computes a health-aware ServiceLevel byte via ServiceLevelCalculator (see below), and forwards it to IServiceLevelPublisher.
IServiceLevelPublisher / SdkServiceLevelPublisher	OtOpcUa.Commons.OpcUa / OtOpcUa.OpcUaServer	Writes the byte into the SDK's Server.ServiceLevel Variable. Production binds DeferredServiceLevelPublisher, which swaps in the real SdkServiceLevelPublisher once the SDK is up (it needs IServerInternal, available only after StandardServer.Start); until then writes route through NullServiceLevelPublisher.
ServiceLevelCalculator	OtOpcUa.Cluster.Redundancy (Core.Cluster)	Pure function (NodeHealthInputs) → byte — the DB/probe-aware tiering (see truth table below). Covered by ServiceLevelCalculatorTests. Now the live publish path — OpcUaPublishActor calls it on every HealthTick and RedundancyStateChanged event. Moved to Core.Cluster so Runtime can reach it without a Runtime→ControlPlane reference.
DbHealthProbeActor	OtOpcUa.Runtime.Health	Runs SELECT 1 against ConfigDb every 5s. Read by the health endpoint AND by OpcUaPublishActor (the DbReachable ServiceLevel input). Spawned only when a ConfigDb is present (admin-role or fused admin+driver nodes) — a driver-only node has no IDbContextFactory<OtOpcUaConfigDbContext> to probe, so this actor is not spawned there at all; see "DB-less nodes" below.
PeerProbeSupervisor	OtOpcUa.Runtime.Health	Per-node; subscribes to the redundancy-state topic and maintains one PeerOpcUaProbeActor child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers.
PeerOpcUaProbeActor	OtOpcUa.Runtime.Health	Spawned by PeerProbeSupervisor; pings a peer opc.tcp://peer:4840 with a TCP connect (2s timeout) and publishes OpcUaProbeResult on the redundancy-state topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe.
ClusterRoleInfo	OtOpcUa.Cluster	Live view of cluster membership + role-leader; exposes IClusterRoleInfo to the rest of the host. (Akka's role-leader is not what elects the redundancy Primary — see below.)

ServiceLevel tiers

Health-aware tiering (ServiceLevelCalculator — live path)

ServiceLevelCalculator.Compute(NodeHealthInputs) is the live publish path. OpcUaPublishActor calls it on every HealthTick (~5 s) and on each RedundancyStateChanged snapshot, then forwards the result through IServiceLevelPublisher to the SDK's Server.ServiceLevel Variable.

The four inputs are sourced locally per driver node:

Input	Source
MemberState	Local SelfMember.Status from the Akka cluster (Up / Joining / Leaving / …).
DbReachable	Local DbHealthProbeActor — OpcUaPublishActor Asks it on each HealthTick; an Ask timeout is treated as Reachable=false.
OpcUaProbeOk	Result of a peer probing THIS node's OPC UA endpoint: PeerProbeSupervisor spawns one PeerOpcUaProbeActor per OTHER driver-role peer; each probe publishes OpcUaProbeResult(probed-node, ok) on the redundancy-state topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → true (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT false demotes.
Stale (derived)	!DbReachable || (now − lastDbHealth.AsOfUtc) > 30 s || (now − snapshotEntry.AsOfUtc) > 30 s.
IsDriverPrimary	The local node's entry in the RedundancyStateChanged snapshot from RedundancyStateActor.

The resulting truth table (all tiers are now reachable at runtime):

Tier	Byte	Condition
Down / Detached	0	Member status is not Up or Joining (leaving, removed, exiting), OR node has no driver role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255.
Critically degraded	100	ConfigDb unreachable AND data is stale.
Stale	200	Data stale but ConfigDb reachable.
Healthy follower	240	DB reachable + OPC UA probe ok + not stale + not the driver Primary.
Healthy primary	250	Same as healthy follower + this node is the driver Primary (+10 bonus).

Secondary 100 → 240 (behavior change). Previously a healthy Secondary published 100 (coarse role-only mapping). It now publishes 240 — both nodes sit at 240/250 under healthy conditions, with the leader still preferred by the +10 bonus. Clients with the standard "pick highest ServiceLevel" heuristic continue to prefer the primary.

DB-less (driver-only) nodes — per-cluster mesh Phase 4

Client-visible change. Per-cluster mesh Phase 4 cut the ConfigDb connection off driver-only nodes (Akka role driver, not admin) — Program.cs registers AddOtOpcUaConfigDb only when hasAdmin. With no ConfigDb to probe, DbHealthProbeActor is not spawned on such a node (ServiceCollectionExtensions.WithOtOpcUaRuntimeActors skips it whenever the resolved IDbContextFactory<OtOpcUaConfigDbContext> is null), and OpcUaPublishActor runs in a dbLess mode instead of the health-aware path above:

• DbReachable is fed constant true — LocalDb is the config store and is in-process, so it can never be "unreachable" the way a remote SQL Server can.
• Stale is derived from the redundancy-state snapshot age alone (`(now − snapshotEntry.AsOfUtc)

  30 s`) — there is no DB-health term to fold in.

• OpcUaProbeOk and IsDriverPrimary are unchanged (peer OPC UA probe, oldest-Up-member election).

Consequence: a healthy driver-only node publishes 240 as a follower or 250 as the driver Primary even when central SQL is unreachable — the survive-alone posture the fetch-and- cache design (Phase 3) exists to support. Contrast this with a Direct/DB-backed node (fused admin+driver, or any node still reading ConfigDb directly): an unreachable ConfigDb there still drives DbReachable=false, which forces Stale=true in the derivation above and the node drops to 100 (or 0 once the sample itself goes stale) via the health-aware tiering, not 240/250. Same byte, opposite meaning depending on which kind of node is publishing it — a client or operator reading ServiceLevel in isolation cannot tell "central is down and I don't care" (driver-only, 240) from "central is down and I'm degraded" (DB-backed, 100/0) without also knowing the node's role shape. ServiceLevelCalculator.Compute itself is unchanged; only the inputs a DB-less node feeds it changed — see OpcUaPublishActor.RecomputeServiceLevel's _dbLess branch.

Backward-compatible fallback (legacy seam)

A node with no DbHealthStatus wired (e.g. early bootstrap window before the first DbHealthProbeActor reply) falls back to the old role-only mapping: Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the first DbHealthStatus arrives the calculator takes over. The first computed ServiceLevel (even 0) is always published so no node lingers at the SDK default 255.

Roles come from RedundancyStateActor.BuildSnapshot: a node with the driver role is Primary when it is the oldest Up member carrying the driver role, otherwise Secondary; a node without the driver role is Detached.

Oldest, not role leader (changed 2026-07-21). The election previously used ClusterState.RoleLeader("driver") — the lowest-addressed Up member with the role. Akka offers both, and they name different nodes: role leader is ordered by address (host, then port), oldest by up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node re-joins as the youngest while keeping its address, so if it holds the lower address it becomes role leader. Meanwhile ClusterSingletonManager places singletons on the oldest, so the old derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by RedundancyPrimaryElectionTests, which forms a real two-node cluster where the oldest node holds the higher address so the two orderings genuinely disagree. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary to stop the name asserting the old derivation.

RESOLVED by Phase 6 — the election is now per application Cluster, not per Akka mesh. Phase 6 of the per-cluster mesh program split the single fleet-wide Akka mesh into one independent 2-node mesh per application Cluster (see Per-cluster meshes (Phase 6) below). RedundancyStateActor is now a cluster-{ClusterId}-scoped singleton spawned on every driver node, so each pair elects its own Primary — the whole-Akka-cluster mismatch this block used to describe no longer exists. See docs/plans/2026-07-24-mesh-phase6-partition-and-colocation.md.

Originally surfaced by the LocalDb Phase 2 live gate, where the old whole-mesh election stopped the alarm-history drain on every node in the fleet — see docs/plans/2026-07-20-localdb-phase2-live-gate.md.

Per-cluster meshes (Phase 6)

The fleet no longer runs one Akka mesh. Phase 6 split it into three independent 2-node meshes — the central pair (roles admin,driver,cluster-MAIN), the site-a pair (driver,cluster-SITE-A), and the site-b pair (driver,cluster-SITE-B) — all sharing the same ActorSystem name (otopcua) on one network, separated purely by per-pair self-first seed partitioning: each node's Cluster:SeedNodes lists only itself and its own pair partner, so gossip never crosses a pair boundary (see Bootstrap: self-first seed ordering).

Consequences:

• One Primary per pair — two or more Primaries fleet-wide, by design. Each mesh runs its own election, so a three-mesh fleet has three Primaries, one per application Cluster. This is correct, not a regression of the old single-Primary assumption.
• RedundancyStateActor is a cluster-{ClusterId}-scoped singleton, spawned on every driver node (falling back to the plain driver role for a legacy node not carrying a cluster role). Each pair elects and publishes its own redundancy-state on its own mesh's DPS — the DPS topic name is unchanged, but it no longer reaches past the pair.
• ClusterNodeAddressReconciler is own-cluster-scoped — an admin node can only reconcile ClusterNode rows belonging to its own cluster; it has no gossip visibility into another mesh to reconcile against.
• CentralCommunicationActor creates one ClusterClient per application Cluster and fans commands across them, rather than one fleet-wide client. Central reaches a site mesh over ClusterClient (commands), gRPC (telemetry — see docs/Telemetry.md), and ConfigServe (config fetch — see ConfigSource/ConfigServe), never over gossip.
• SplitTopologyTransportValidator fails host start on a node carrying a cluster-{ClusterId} role unless it is also configured for the split-safe transports: MeshTransport:Mode = ClusterClient, Telemetry:Mode = Grpc (driver nodes), TelemetryDial:Mode = Grpc (admin nodes). A DPS-mode transport on a cluster-role node would silently try to gossip across a partition that no longer exists, so the validator fails closed instead. Legacy/non-cluster-role nodes are exempt.
• The compiled transport-mode defaults were deliberately NOT flipped — MeshTransport:Mode, Telemetry:Mode, and TelemetryDial:Mode all still default to Dps. A blast-radius assessment found flipping the compiled default would break every integration test and every existing appsettings.json profile that doesn't explicitly set these keys. The guardrail is the validator above plus explicit per-deployment configuration, not a default flip.

Secrets replication is pair-local

Akka DPS secret replication is scoped to each 2-node mesh: a pair replicates its own secrets to its own partner and no further. There is no fleet-wide or cross-cluster secret sync after the split — by design, consistent with Phase 4 (site nodes have no SQL, so there is no SQL-backed hub to fan a cross-mesh sync through).

Data flow

Cluster topology event ──────────────────────────────────────────┐
                                                                   ▼
                                               RedundancyStateActor (admin singleton)
                                                                   │  debounce 250ms
                                                                   ▼
                                               DPS topic "redundancy-state"
                                                    │                         ▲
                            ┌───────────────────────┘                         │
                            │                                                  │
                            ▼                                                  │
              Driver node: OpcUaPublishActor                                   │
              ┌─────────────────────────────────────────────────────────┐      │
              │  Inputs collected per ~5s HealthTick:                   │      │
              │   • MemberState  ← Akka SelfMember.Status               │      │
              │   • DbReachable  ← DbHealthProbeActor (Ask, timeout→F) │      │
              │   • OpcUaProbeOk ← OpcUaProbeResult about THIS node    │──────┘
              │   • Stale        ← derived from above timestamps        │  PeerProbeSupervisor
              │   • IsLeader     ← RedundancyStateChanged snapshot      │  → PeerOpcUaProbeActor(s)
              │                                                         │  publish OpcUaProbeResult
              │  ServiceLevelCalculator.Compute(NodeHealthInputs)       │  on "redundancy-state"
              │  → byte (0/100/200/240/250)                             │
              └───────────────────────────────────────────────────────-─┘
                            │
                            ▼
              IServiceLevelPublisher (SdkServiceLevelPublisher)
                            │
                            ▼
              OPC UA Server.ServiceLevel Variable

Both DbHealthProbeActor and PeerOpcUaProbeActor feed the live publish path. The peer probe publishes OpcUaProbeResult on the redundancy-state topic; OpcUaPublishActor consumes only results whose target is itself and applies freshness-debouncing before passing them to the calculator. DbHealthProbeActor is queried directly via Ask on each HealthTick.

The admin singleton is the cluster's only RedundancyStateActor. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current Cluster.State. There is no DB-persisted state to recover.

Configuration

Per-node identity comes from appsettings.json + the OTOPCUA_ROLES env var:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    // Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
    // node-b.lan's own config lists node-b.lan first and node-a.lan second.
    "SeedNodes": [
      "akka.tcp://otopcua@node-a.lan:4053",
      "akka.tcp://otopcua@node-b.lan:4053"
    ],
    "Roles": ["admin", "driver"]
  }
}

OTOPCUA_ROLES=admin,driver

Both nodes share the same ConfigDb connection string; Cluster.PublicHostname + Roles are what makes them distinct in cluster gossip. List both peers in SeedNodes on both nodes so either can cold-start alone — and list each node ITSELF first: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See Bootstrap: self-first seed ordering.

There is no longer a Node:NodeId setting and no ClusterNode.RedundancyRole column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as host:port of the cluster PublicHostname (see ClusterRoleInfo.LocalNode for the formula).

RedundancyStateActor NodeId consistency (fixed). RedundancyStateActor now keys each node's NodeRedundancyState entry by the canonical host:port node id (via a ToNodeId(Address) helper mirroring ClusterRoleInfo.ToNodeId). Previously it keyed by member.Address.Host (host-only, e.g. central-2); since every subscriber matches by the canonical host:port form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes RedundancyStateActor consistent with the stated contract above. Additionally, RedundancyStateActor now re-publishes the current snapshot on a periodic heartbeat (default 10 s) so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).

The ClusterNode.ServiceLevelBase column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by ServiceLevelCalculator from cluster role and live health inputs, independent of this stored preference.

Peer URI advertising

Each node advertises its partner via OpcUaApplicationHostOptions.PeerApplicationUris (an IList<string>, default empty). OpcUaApplicationHost.PopulateServerArray appends each configured peer URI to the SDK's IServerInternal.ServerUris string table after server startup, so that Server.ServerArray reads served by OnReadServerArray return both self + peers. The options bind from the OpcUa config section (see Program.cs — AddValidatedOptions<OpcUaApplicationHostOptions>(…, "OpcUa")). Set this per-node in appsettings.json:

{
  "OpcUa": {
    "PeerApplicationUris": ["urn:node-b:OtOpcUa"]
  }
}

Node A lists Node B's ApplicationUri and vice-versa. Validated by DualEndpointTests in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ — boots two OpcUaApplicationHost instances on loopback, asserts a real OPCFoundation client Session reading Server.ServerArray from Node A sees both URIs.

Split-brain / downing

The default downing strategy is auto-down (changed 2026-07-21). It is selected by Cluster:SplitBrainResolverStrategy, and ServiceCollectionExtensions.BuildDowningHocon turns that into the akka.cluster.downing-provider-class actually installed:

Cluster:SplitBrainResolverStrategy	Provider installed	Posture
auto-down (default)	Akka.Cluster.AutoDowning with auto-down-unreachable-after = 15s	Availability. The leader among the reachable members downs the unreachable peer, so a crash of either node — oldest included — fails over in place.
keep-oldest	Akka.Cluster.SBR.SplitBrainResolverProvider reading the split-brain-resolver block in akka.conf	Partition-safety. A partition can never run dual-active. Not safe for a 2-node pair — see below.

Any other value fails the host at startup rather than falling through to a default.

Why keep-oldest is no longer the default

The previous documentation in this section — and the code comments behind it — asserted that keep-oldest with down-if-alone = on was "the correct strategy for a 2-node warm-redundancy pair." That was wrong, and the error was not conservative: it described a total-outage configuration as the recommended one.

In Akka.NET 1.5.62's KeepOldest.OldestDecision, the down-if-alone rescue branch requires the surviving side to hold ≥ 2 members. In a two-node cluster a lost peer is always a 1-vs-1 split, so the branch never fires and the lone survivor falls through to DownReachable — downing itself — and run-coordinated-shutdown-when-down = on then terminates it. down-if-alone is a 3+-node feature; it does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members. See ../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md for the upstream decision record, which OtOpcUa follows.

The alternatives do not help a pair: static-quorum with quorum 1 hits Akka's IsTooManyMembers guard and returns DownAll; with quorum 2 the survivor downs itself; keep-majority keeps the lowest-address side, which just moves the fatal crash from "the oldest node" to "the other node"; lease-majority needs a shared lease store both nodes can reach.

The accepted trade

Under auto-down, a genuine network partition (both nodes alive, link cut) leaves each side downing the other and continuing alone: both run active, both hold singletons, and both advertise the primary ServiceLevel. Recovery is operator-driven — after the link heals, restart one side; it rejoins as a fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual downing quarantines the association.

This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and a crash under keep-oldest was unrecoverable without operator action anyway.

Deployments that would rather take an outage than ever run dual-active can set Cluster:SplitBrainResolverStrategy: "keep-oldest" — but should read the section above first, because for a two-node pair that choice means any crash of the oldest node is a full outage.

Recovery still depends on supervision

1. A service supervisor that restarts the process on exit — production Install-Services.ps1 sets sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000; the docker-dev rig sets restart: unless-stopped.
2. The recovery watchdog ActorSystemTerminationWatchdog (registered in Program.cs after AddAkka) — it watches ActorSystem.WhenTerminated and, on an unexpected self-down, calls IHostApplicationLifetime.StopApplication() so the process exits and the supervisor restarts it instead of idling forever with a dead actor system. Under auto-down the survivor no longer needs this (it is never the one downed); the downed node still does.
3. Both peers listed in SeedNodes on every node, each node listing itself first, so a restarted node can re-join via either peer and a node cold-starting while its peer is dead still forms the cluster. Without the self-first ordering it waits in InitJoin forever (auto-down removes the crash outage, not that one) — see Bootstrap: self-first seed ordering.

On HardKillFailoverTests: that in-process test hard-kills the oldest and asserts the survivor becomes sole driver role-leader. It passed even under keep-oldest — because it simulates the crash with provider.Transport.Shutdown(), which leaves node A's ActorSystem alive (a transport partition with the oldest still running), not a real process death. A real 2-container docker kill of the oldest downed the survivor (verified 2026-07-15, archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md). It is a standing example of a green test over a fatal defect: keep the distinction between a transport partition and a process death in mind when reading it.

SplitBrainResolverActivationTests pins the strategy by starting a real host through the production bootstrap and reading akka.cluster.downing-provider-class back off the running ActorSystem — asserting the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the losing one still looks correct in the file.

Failover during a partition is what the cluster and supervisor do automatically. For a planned swap — patching, draining a node — see Manual failover below.

Manual failover

The cluster redundancy page (/clusters/{id}/redundancy) carries a Live redundancy panel — the current driver Primary and every Up driver member, read from live cluster state — and an admin-gated Trigger failover button.

The mechanism is a graceful cluster Leave of the current Primary, never a Down: IManualFailoverService.FailOverDriverPrimaryAsync selects the oldest Up driver member — the identical query to RedundancyStateActor.SelectDriverPrimary, pinned together by ManualFailoverServiceTests.Parity_with_SelectDriverPrimary so the node acted on is exactly the node the election names. The leaving node hands its singletons over through the cluster-leave phases, CoordinatedShutdown runs, ActorSystemTerminationWatchdog exits the process, the supervisor restarts it, and it rejoins as the youngest member — so the survivor is now the oldest, becomes Primary, and advertises ServiceLevel 250. OPC UA clients re-select.

Rule	Behaviour
Authorization	AdminUiPolicies.FleetAdmin (Administrator only) — gated in markup and re-checked server-side before the call. Deliberately not ConfigEditor, which the neighbouring cluster pages use: this restarts a production node rather than editing config, and ConfigEditor also admits Designer.
Peer guard	Disabled, with the reason as its tooltip, when fewer than two Up driver members exist — with one node a "failover" is a shutdown. Re-evaluated inside the service against live state, so a stale page cannot bypass it.
Confirmation	A dialog naming the node, and what follows (restart, ServiceLevel 250 moves, clients re-select).
Audit	Written through the shared ZB.MOM.WW.Audit.IAuditWriter seam before the Leave — action cluster.manual-failover, actor, target. A refused failover changes nothing and writes nothing.

Phase 6 of the mesh program (see Per-cluster meshes (Phase 6)) split the fleet into one 2-node mesh per application Cluster, so the button now acts on this pair's own Primary — there is no longer a whole-mesh vs. per-cluster ambiguity to caveat.

Live gate outstanding. The auto-down change is verified by unit tests against the effective configuration, and by the sister project's live drill on an equivalent Akka version and topology. It was not drillable on an OtOpcUa rig before Phase 6, because the docker-dev rig ran a single six-node mesh where the 1-vs-1 pathology cannot occur. Phase 6 has landed and every mesh is now exactly two nodes, so the drill (kill the oldest container of a genuine two-node cluster; assert the survivor stays up, takes the singletons, and reports the primary ServiceLevel) is finally testable — deferred to Phase 7's failover drill. See docs/plans/2026-07-21-per-cluster-mesh-design.md §6.2 / Phase 0a.

Bootstrap: self-first seed ordering

Listing both peers in SeedNodes does not mean either node can cold-start alone — and neither does any downing strategy: auto-down removes the crash outage, not this one. Akka runs a different bootstrap process depending on whether seed-nodes[0] is the node's own address:

seed-nodes[0] is…	Process Akka runs	Can it form a NEW cluster?
this node's own address	FirstSeedNodeProcess	Yes — it InitJoins the other seeds and self-joins once seed-node-timeout (5 s) passes with nobody answering.
some other node	JoinSeedNodeProcess	No — it retries InitJoin forever, however long the peer stays down.

So the ordering is the mechanism. Every node that is a seed of its own mesh lists ITSELF first and its partner second (decision 2026-07-22):

// central-1                                     // central-2
"SeedNodes": [                                   "SeedNodes": [
  "akka.tcp://otopcua@central-1:4053",             "akka.tcp://otopcua@central-2:4053",
  "akka.tcp://otopcua@central-2:4053"              "akka.tcp://otopcua@central-1:4053"
]                                                ]

The rule is conditional: it binds only when a node's own address appears in its own seed list. A driver-only site node is seeded solely by central-1 (today's docker-dev topology) — it is legitimately not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every site node in the fleet.

AkkaClusterOptionsValidator enforces it at startup (AddValidatedOptions → ValidateOnStart in AddOtOpcUaCluster), because the invariant otherwise fails silently: the process runs, the port listens, the node simply never becomes a member. Identity is compared on Cluster:PublicHostname (falling back to Cluster:Hostname when blank) and Cluster:Port — the address Akka puts in SelfAddress. Comparing against Cluster:Hostname would be worse than wrong: in docker-dev it is 0.0.0.0, which matches no seed URI anywhere, so the rule would silently exempt the entire rig.

Sequential recovery is island-free by construction: a FirstSeedNodeProcess node self-joins only when no seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both nodes cold-starting simultaneously converge on one cluster while they are mutually reachable — the InitJoin handshake resolves it before either self-join deadline expires. The residual risk is a boot-time partition (both cold-starting, mutually unreachable): both form, which is the same dual-active class the auto-down strategy already accepts, with the same recovery (restart one side).

Pinned by SelfFirstSeedBootstrapTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/), which starts real in-process clusters through the production bootstrap at production failure-detection timings: a lone cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting together converge on one cluster. Its falsifiability control — the old peer-first ordering, same scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control self-join proving the node was formable all along. AkkaClusterOptionsValidatorTests covers the validator, including the site-node exemption and the 0.0.0.0 identity trap.

Why the self-form watchdog was retired (2026-07-22)

The first fix for this gap was Cluster:SelfFormAfter + ClusterBootstrapFallback: wait 10 s for membership, then call Cluster.Join(SelfAddress). Both are deleted. A timer sits outside Akka's join handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" — and Cluster.Join(SelfAddress) is not ignored mid-handshake: it wins.

Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence). The watchdog islanded a node that manual failover had bounced. central-1 restarted, received InitJoinAck from central-2 at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window, because the peer's ring still held its previous incarnation (Exiting → Down → Removed, retired at 10:49:06). At 10:49:15 the watchdog fired and the node formed a second cluster, islanded until an operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of that plan collided. A TCP reachability guard (ea45ace1) patched that one shape — never self-form while a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely passed: central-2 cold-started alone in 10 s and a non-seed site-a-2 correctly stayed out.

The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because the decision is made inside the handshake, by Akka, on the actual InitJoin replies. It was also inert for every non-seed node, so after this change it could never usefully fire. SelfFormBootstrapTests is replaced by SelfFirstSeedBootstrapTests, whose Restarting_node_rejoins_its_live_peer_instead_of_islanding is the drilled scenario above.

Live gate outstanding for the new mechanism. Self-first ordering is verified here by in-process clusters at production timings, and on the sister project (ScadaBridge 4a6341d8) on real clusters and live docker containers. It has not been re-drilled on the OtOpcUa docker-dev rig since the swap — the central-2 cold-start-alone drill should be repeated on the rebuilt image. Owned by docs/plans/2026-07-22-per-cluster-mesh-program.md Phase 7 alongside the auto-down 1-vs-1 gate.

Primary data-plane gate (writes, acks, alerts emit)

Three data-plane surfaces are Primary-gated so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator writes (DriverHostActor.HandleRouteNodeWrite), native-alarm acks (HandleRouteNativeAlarmAck), and the native/scripted alerts emit (ForwardNativeAlarm / ScriptedAlarmHostActor.OnEngineEmission). All three route through one policy — PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount) (OtOpcUa.Runtime.Drivers):

Local role (last redundancy snapshot)	Cluster driver members	Decision
Primary	any	service
Secondary / Detached	any	deny
unknown (no snapshot yet, or snapshot omitted this node)	≤ 1 (single-node / boot on a lone driver)	service (boot-window / single-node posture)
unknown	≥ 2 (a real driver peer exists)	deny (default-deny — a Primary peer exists; don't act until a snapshot proves this node is it)

Before archreview 03/S4 the unknown-role case defaulted to allow unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s RedundancyStateActor heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a multi-driver cluster now default-denies. The driver member count is read live from Cluster.State.Members (Up members with the driver role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), DriverHostActor logs a one-time Warning so the silent mismatch is diagnosable.

What a client sees during the boot window (multi-node, role unknown): an inbound write is rejected (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 AuditWriteUpdateEvent; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window ("not primary (role unknown)") from a steady-state Secondary ("not primary"). Acks are dropped (Warning-logged when role-unknown). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.

Under warm/hot redundancy both cluster nodes run ScriptedAlarmHostActor and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on /alerts and duplicate historian writes, only the Primary node publishes externally:

• alerts topic emission — ScriptedAlarmHostActor and DriverHostActor.ForwardNativeAlarm subscribe to the redundancy-state DPS topic and cache the local node's RedundancyRole, then gate the cluster alerts publish through PrimaryGatePolicy (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain ungated on both nodes so the secondary is always ready to serve clients after a failover.
• HistorianAdapterActor historization — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the alerts DPS topic and translates each AlarmTransitionEvent → AlarmHistorianEvent before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
• The alarm sink's DRAIN — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. LocalDbStoreAndForwardSink takes a drainGate fed from IRedundancyRoleView — a singleton DriverHostActor publishes its PrimaryGatePolicy verdict to, because the drain runs on a timer and cannot receive RedundancyStateChanged itself. A gated-off node reports HistorianDrainState.NotPrimary. Delivery is at-least-once across a failover by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See AlarmHistorian.md.

Net effect: each alarm transition appears once on /alerts and would historize once, not once per node.

See ScriptedAlarms.md and AlarmTracking.md for the scripted-alarm engine internals.

Command transport (central↔node)

Everything above rides one Akka mesh: central publishes on DistributedPubSub, every node subscribes, and the two ends must be members of the same cluster. Per-cluster mesh Phase 2 adds a second transport under MeshTransport:Mode (see Configuration.md § MeshTransport) that does not need shared membership — the prerequisite for splitting the fleet into one mesh per application Cluster.

	Dps (default)	ClusterClient
Central → node commands	Publish on deployments / driver-control / alarm-commands	ClusterClient.SendToAll to /user/node-communication, re-emitted on the node's local EventStream
Node → central acks	Publish on deployment-acks	ClusterClient.Send to /user/central-communication, forwarded to the deploy-coordinator singleton
Requires shared cluster membership	yes	no

Two properties are load-bearing and easy to break:

• Both comm actors are registered per node, NOT as cluster singletons. A ClusterClient rotates across its contact points and must find a live comm actor at whichever node answers. As a singleton the boundary would be reachable through exactly one node, and rotation would fail silently against the others — acks landing or not depending on where the client happened to settle. akka.cluster.client.receptionist.role is deliberately empty for the same reason.
• Inbound commands go to the node-local EventStream, never back onto a DPS topic. DPS is mesh-wide and central SendToAlls to every node, so a DPS re-publish would deliver N copies of every command to every subscriber.

Phase 2 changes no redundancy semantics: the Primary gate, ServiceLevel, and the singleton layout are untouched, and redundancy-state itself stays on DPS (it is bidirectional and built from Cluster.State, which makes it genuinely mesh-bound — the hardest remaining dependency before the meshes can split).

Observability telemetry took the opposite path (Phase 5). The four live-panel channels (alerts, script-logs, driver-health, driver-resilience-status) moved off DPS onto a dedicated gRPC server-streaming contract — each driver node hosts, central dials in — selected by Telemetry:Mode/TelemetryDial:Mode (default Dps). redundancy-state is deliberately not part of that migration: it stays on DPS in every mode because it is pair-local and Cluster.State-built, not a fleet-wide observability broadcast. See docs/Telemetry.md for the full architecture and Configuration.md § Telemetry/TelemetryDial for the config keys.

Config bytes travel out-of-band (Phase 3). Commands stay on one of the two transports above, but the configuration artifact itself does not ride any Akka message — DispatchDeployment is payload-free (DeploymentId + RevisionHash). Under ConfigSource:Mode = FetchAndCache (see Configuration.md § ConfigSource) a driver node fetches the bytes from central over a dedicated h2c gRPC stream, verifies SHA-256 == RevisionHash, and reads only its LocalDb cache — no central-SQL config read. Both pair nodes fetch independently, exactly as both read SQL today: there is no Primary gating of the fetch. That is safe because StoreAsync is idempotent and the cache replicates (pair replication carries the artifact to a node that missed its own fetch), and it keeps config delivery decoupled from redundancy timing. Central serves the bytes from its SQL row; a FetchAndCache node that cannot reach central fails the apply and keeps last-known-good (it never tears down to an empty address space), and it boots its served state from the LocalDb pointer with no central dependency at all.

Pair-local store (LocalDb — Phases 1 + 2)

Independently of the ServiceLevel machinery above, every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database. Phase 2 added the alarm store-and-forward buffer (alarm_sf_events) to it alongside the config cache, so a node that dies holding undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above and AlarmHistorian.md. Phase 1 caches the deployed-configuration artifact (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver node restarting into a central-SQL-Server outage would come up with no configuration at all. With the cache, it boots from the last artifact it applied instead, and logs a running-from-cache signal.

• What replicates. The cache can optionally replicate to the node's redundant pair peer over the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate: deployment_artifacts and deployment_pointer. Replication is default-OFF and fail-closed — it stays inert until LocalDb:SyncListenPort + a peer are configured, and a missing/mismatched LocalDb:Replication:ApiKey refuses or silently halts sync. It is enabled per-pair as an opt-in.
• The replicated-cache payoff. In a pair with replication on, either node can be the one that applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a central outage can boot the current config even if it never applied that deploy itself — its peer did, and the row replicated.
• What it does NOT cover. The cache is a boot fallback for central-SQL outages only, not a standby data path and not a replacement for central. A new deployment still requires central SQL; the cache is read only when the central fetch fails at startup, and the node resumes fresh-config behavior once central returns. It does not replicate live tag values, alarms, or historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate logic: a node running from cache still participates in redundancy normally.

Full detail: the enablement runbook at docs/operations/2026-07-20-localdb-pair-replication.md and the LocalDb section of docs/Configuration.md.

Client-side failover

The OtOpcUa Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI supports -F / --failover-urls for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See Client.CLI.md.

Observability

OpcUaPublishActor emits one metric on every ServiceLevel transition (it suppresses no-op repeats of the same byte):

Metric	Type	Notes
otopcua.redundancy.service_level_change	Counter ({change})	OPC UA Server.ServiceLevel transitions emitted by the redundancy state. Tagged with level = the new byte.
otopcua.redundancy.primary_gate_denied	Counter ({denial})	Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged site = write | ack | alarm-emit and reason = secondary | detached | role-unknown. A non-zero role-unknown rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering).
otopcua.opcua.apply.failed	Counter ({apply})	Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged kind = rebuild (the rebuild threw) | nodes (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — OpcUaPublishActor also logs the degraded apply at Error.
galaxy.writes.advise_failed	Counter ({write})	Galaxy writes short-circuited to Bad because AdviseSupervisory failed — the value could not have committed, so the write fails closed (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter ZB.MOM.WW.OtOpcUa.Driver.Galaxy.
galaxy.writes.unconfirmed	Counter ({write})	Galaxy writes reported Good off an empty gateway statuses array (command accepted; the COM-side commit is unconfirmed pending the mxaccessgw WriteComplete correlation follow-up). The honest signal for the optimistic-Good rate until that cross-repo fix lands. On the Galaxy driver meter.

The redundancy/apply meters are defined on OtOpcUaTelemetry (src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs); the Galaxy write meters hang off the driver's own meter (EventPump.MeterName). All surface through whatever OpenTelemetry exporter the host configures.

Galaxy fail-closed write semantics (archreview 06/S-1): a non-secured Galaxy write commits only when its item is AdviseSupervisory-advised first (the writer runs with no login). If the advise fails, the write is not issued — GatewayGalaxyDataWriter returns BadCommunicationError (a knowingly-lost write is never reported as Good). An empty statuses array on the write reply is still treated as Good but metered as galaxy.writes.unconfirmed. See Historian.md / the Galaxy driver notes for the write path.

Depth reference

For the full design — message contracts, tiered calculator truth table, recovery semantics — see docs/plans/2026-05-26-akka-hosting-alignment-design.md §6.