Files

T

Joseph Doherty 983310edbb docs(redundancy): record the self-form fallback live gate as PASSED on docker-dev

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW

2026-07-22 06:35:06 -04:00

36 KiB

Raw Blame History

Redundancy (v2)

Overview

OtOpcUa supports OPC UA non-transparent warm/hot redundancy. Two or more OtOpcUa.Host processes run side-by-side, share the same Config DB, and join the same Akka.NET cluster. Each process owns a distinct ApplicationUri; OPC UA clients discover both endpoints by reading Server.ServerArray (NodeId i=2254) on either node and pick one based on the ServiceLevel byte that each server publishes.

Discovery surface. The ServerArray path on the Server object is what each node populates with self + peer ApplicationUris — see OpcUaApplicationHost.PopulateServerArray and the per-node PeerApplicationUris option below. The redundancy-object-type ServerUriArray proper (a child of Server.ServerRedundancy) remains deferred pending an SDK object-type upgrade; clients should read Server.ServerArray for peer discovery today.

v2 change. v1's operator-managed ClusterNode.RedundancyRole column + RedundancyCoordinator / ApplyLeaseRegistry / PeerHttpProbeLoop are gone. Primary/secondary is now derived from Akka cluster membership — the oldest Up member carrying the driver role. The operator no longer writes a role into the DB; cluster topology drives ServiceLevel automatically.

The runtime pieces live in:

Component	Project	Role
`RedundancyStateActor`	`OtOpcUa.ControlPlane.Redundancy`	Admin-role cluster singleton; subscribes to cluster topology events, debounces 250ms, broadcasts `RedundancyStateChanged` on the `redundancy-state` DPS topic.
`OpcUaPublishActor`	`OtOpcUa.Runtime.OpcUa`	Per-driver-node; subscribes to the `redundancy-state` topic, computes a health-aware ServiceLevel byte via `ServiceLevelCalculator` (see below), and forwards it to `IServiceLevelPublisher`.
`IServiceLevelPublisher` / `SdkServiceLevelPublisher`	`OtOpcUa.Commons.OpcUa` / `OtOpcUa.OpcUaServer`	Writes the byte into the SDK's `Server.ServiceLevel` Variable. Production binds `DeferredServiceLevelPublisher`, which swaps in the real `SdkServiceLevelPublisher` once the SDK is up (it needs `IServerInternal`, available only after `StandardServer.Start`); until then writes route through `NullServiceLevelPublisher`.
`ServiceLevelCalculator`	`OtOpcUa.Cluster.Redundancy` (`Core.Cluster`)	Pure function `(NodeHealthInputs) → byte` — the DB/probe-aware tiering (see truth table below). Covered by `ServiceLevelCalculatorTests`. Now the live publish path — `OpcUaPublishActor` calls it on every `HealthTick` and `RedundancyStateChanged` event. Moved to `Core.Cluster` so Runtime can reach it without a Runtime→ControlPlane reference.
`DbHealthProbeActor`	`OtOpcUa.Runtime.Health`	Per-node; runs `SELECT 1` against ConfigDb every 5s. Read by the health endpoint AND by `OpcUaPublishActor` (the `DbReachable` ServiceLevel input).
`PeerProbeSupervisor`	`OtOpcUa.Runtime.Health`	Per-node; subscribes to the `redundancy-state` topic and maintains one `PeerOpcUaProbeActor` child per OTHER driver-role peer (spawn on join, stop on departure), so every node is continuously probed by its peers.
`PeerOpcUaProbeActor`	`OtOpcUa.Runtime.Health`	Spawned by `PeerProbeSupervisor`; pings a peer `opc.tcp://peer:4840` with a TCP connect (2s timeout) and publishes `OpcUaProbeResult` on the `redundancy-state` topic. A full secure-channel Hello handshake is a possible future upgrade; the TCP connect is the current real probe.
`ClusterRoleInfo`	`OtOpcUa.Cluster`	Live view of cluster membership + role-leader; exposes `IClusterRoleInfo` to the rest of the host. (Akka's role-leader is not what elects the redundancy Primary — see below.)

ServiceLevel tiers

Health-aware tiering (`ServiceLevelCalculator` — live path)

ServiceLevelCalculator.Compute(NodeHealthInputs) is the live publish path. OpcUaPublishActor calls it on every HealthTick (~5 s) and on each RedundancyStateChanged snapshot, then forwards the result through IServiceLevelPublisher to the SDK's Server.ServiceLevel Variable.

The four inputs are sourced locally per driver node:

Input	Source
`MemberState`	Local `SelfMember.Status` from the Akka cluster (Up / Joining / Leaving / …).
`DbReachable`	Local `DbHealthProbeActor` — `OpcUaPublishActor` Asks it on each `HealthTick`; an Ask timeout is treated as `Reachable=false`.
`OpcUaProbeOk`	Result of a peer probing THIS node's OPC UA endpoint: `PeerProbeSupervisor` spawns one `PeerOpcUaProbeActor` per OTHER driver-role peer; each probe publishes `OpcUaProbeResult(probed-node, ok)` on the `redundancy-state` topic; the publish actor consumes only results whose target is itself. Freshness-debounced: absent or stale (>30 s) → `true` (benefit of the doubt — single-node clusters and a departed peer never demote); only an actively-observed RECENT `false` demotes.
`Stale` (derived)	`!DbReachable \|\| (now − lastDbHealth.AsOfUtc) > 30 s \|\| (now − snapshotEntry.AsOfUtc) > 30 s`.
`IsDriverPrimary`	The local node's entry in the `RedundancyStateChanged` snapshot from `RedundancyStateActor`.

The resulting truth table (all tiers are now reachable at runtime):

Tier	Byte	Condition
Down / Detached	0	Member status is not `Up` or `Joining` (leaving, removed, exiting), OR node has no `driver` role (Detached). Published immediately — a starting or detached node never leaves the SDK default 255.
Critically degraded	100	ConfigDb unreachable AND data is stale.
Stale	200	Data stale but ConfigDb reachable.
Healthy follower	240	DB reachable + OPC UA probe ok + not stale + not the driver Primary.
Healthy primary	250	Same as healthy follower + this node is the driver Primary (+10 bonus).

Secondary 100 → 240 (behavior change). Previously a healthy Secondary published 100 (coarse role-only mapping). It now publishes 240 — both nodes sit at 240/250 under healthy conditions, with the leader still preferred by the +10 bonus. Clients with the standard "pick highest ServiceLevel" heuristic continue to prefer the primary.

Backward-compatible fallback (legacy seam)

A node with no DbHealthStatus wired (e.g. early bootstrap window before the first DbHealthProbeActor reply) falls back to the old role-only mapping: Primary-leader → 240, Primary → 200, Secondary → 100, Detached → 0. Once the first DbHealthStatus arrives the calculator takes over. The first computed ServiceLevel (even 0) is always published so no node lingers at the SDK default 255.

Roles come from RedundancyStateActor.BuildSnapshot: a node with the driver role is Primary when it is the oldest Up member carrying the driver role, otherwise Secondary; a node without the driver role is Detached.

Oldest, not role leader (changed 2026-07-21). The election previously used ClusterState.RoleLeader("driver") — the lowest-addressed Up member with the role. Akka offers both, and they name different nodes: role leader is ordered by address (host, then port), oldest by up-number. They agree on a freshly-formed cluster, and diverge after any restart — a restarted node re-joins as the youngest while keeping its address, so if it holds the lower address it becomes role leader. Meanwhile ClusterSingletonManager places singletons on the oldest, so the old derivation could name a Primary that was not hosting the singletons, enabling the Primary-gated data plane (writes, alarm acks, the alerts emit, the alarm-history drain) on the wrong node. Pinned by RedundancyPrimaryElectionTests, which forms a real two-node cluster where the oldest node holds the higher address so the two orderings genuinely disagree. NodeRedundancyState.IsRoleLeaderForDriver was renamed IsDriverPrimary to stop the name asserting the old derivation.

KNOWN LIMITATION — the election is per Akka cluster, not per application Cluster. The election yields exactly one Primary across the whole Akka cluster. A fleet that runs several application clusters (each with its own redundant pair) inside one Akka cluster — which is what docker-dev/docker-compose.yml models, with MAIN, SITE-A and SITE-B — therefore has one Primary among all its driver nodes, not one per pair. Every consumer of the role inherits this: ServiceLevel, the alerts emit gate, and the inbound device-write gate below.

The unit of redundancy everywhere else in the product is the application Cluster (ClusterId) — it owns drivers, devices and ClusterNode rows, and the LocalDb deployment-artifact cache is already keyed by it so that a pair shares one entry. Scoping the election the same way is the fix; RedundancyStateActor is an admin-role singleton in the same assembly as AdminOperationsActor, which already reads and groups ClusterNodes by cluster.

Surfaced by the LocalDb Phase 2 live gate, where it stopped the alarm-history drain on every node in the fleet — see docs/plans/2026-07-20-localdb-phase2-live-gate.md. The alarm drain no longer depends on this (it defers only to a node that shares its queue), but the other three gates still do.

Data flow

Cluster topology event ──────────────────────────────────────────┐
                                                                   ▼
                                               RedundancyStateActor (admin singleton)
                                                                   │  debounce 250ms
                                                                   ▼
                                               DPS topic "redundancy-state"
                                                    │                         ▲
                            ┌───────────────────────┘                         │
                            │                                                  │
                            ▼                                                  │
              Driver node: OpcUaPublishActor                                   │
              ┌─────────────────────────────────────────────────────────┐      │
              │  Inputs collected per ~5s HealthTick:                   │      │
              │   • MemberState  ← Akka SelfMember.Status               │      │
              │   • DbReachable  ← DbHealthProbeActor (Ask, timeout→F) │      │
              │   • OpcUaProbeOk ← OpcUaProbeResult about THIS node    │──────┘
              │   • Stale        ← derived from above timestamps        │  PeerProbeSupervisor
              │   • IsLeader     ← RedundancyStateChanged snapshot      │  → PeerOpcUaProbeActor(s)
              │                                                         │  publish OpcUaProbeResult
              │  ServiceLevelCalculator.Compute(NodeHealthInputs)       │  on "redundancy-state"
              │  → byte (0/100/200/240/250)                             │
              └───────────────────────────────────────────────────────-─┘
                            │
                            ▼
              IServiceLevelPublisher (SdkServiceLevelPublisher)
                            │
                            ▼
              OPC UA Server.ServiceLevel Variable

Both DbHealthProbeActor and PeerOpcUaProbeActor feed the live publish path. The peer probe publishes OpcUaProbeResult on the redundancy-state topic; OpcUaPublishActor consumes only results whose target is itself and applies freshness-debouncing before passing them to the calculator. DbHealthProbeActor is queried directly via Ask on each HealthTick.

The admin singleton is the cluster's only RedundancyStateActor. If the admin leader fails over, the new admin node spins up its replacement, re-subscribes to cluster events, and publishes a fresh snapshot from the current Cluster.State. There is no DB-persisted state to recover.

Configuration

Per-node identity comes from appsettings.json + the OTOPCUA_ROLES env var:

{
  "Cluster": {
    "Hostname": "0.0.0.0",
    "Port": 4053,
    "PublicHostname": "node-a.lan",
    "SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
    "Roles": ["admin", "driver"]
  }
}

OTOPCUA_ROLES=admin,driver

Both nodes share the same ConfigDb connection string; Cluster.PublicHostname + Roles are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in SeedNodes); the second node joins via the same SeedNodes list. List both peers in SeedNodes on both nodes so either can cold-start alone — see Bootstrap: the self-form fallback for why the seed list alone is not enough, and what Cluster:SelfFormAfter adds.

There is no longer a Node:NodeId setting and no ClusterNode.RedundancyRole column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as host:port of the cluster PublicHostname (see ClusterRoleInfo.LocalNode for the formula).

RedundancyStateActor NodeId consistency (fixed). RedundancyStateActor now keys each node's NodeRedundancyState entry by the canonical host:port node id (via a ToNodeId(Address) helper mirroring ClusterRoleInfo.ToNodeId). Previously it keyed by member.Address.Host (host-only, e.g. central-2); since every subscriber matches by the canonical host:port form, the mismatch silently meant no node ever matched its own entry — all nodes stayed at the default ServiceLevel 255 and never learned their role. This fix makes RedundancyStateActor consistent with the stated contract above. Additionally, RedundancyStateActor now re-publishes the current snapshot on a periodic heartbeat (default 10 s) so any node that subscribes after the last topology-change publish converges within the interval (DistributedPubSub does not replay to late subscribers).

The ClusterNode.ServiceLevelBase column still exists and is editable in the Admin UI (NodeEdit / Cluster Redundancy pages), but it no longer drives the runtime ServiceLevel — that value is computed by ServiceLevelCalculator from cluster role and live health inputs, independent of this stored preference.

Peer URI advertising

Each node advertises its partner via OpcUaApplicationHostOptions.PeerApplicationUris (an IList<string>, default empty). OpcUaApplicationHost.PopulateServerArray appends each configured peer URI to the SDK's IServerInternal.ServerUris string table after server startup, so that Server.ServerArray reads served by OnReadServerArray return both self + peers. The options bind from the OpcUa config section (see Program.cs — AddValidatedOptions<OpcUaApplicationHostOptions>(…, "OpcUa")). Set this per-node in appsettings.json:

{
  "OpcUa": {
    "PeerApplicationUris": ["urn:node-b:OtOpcUa"]
  }
}

Node A lists Node B's ApplicationUri and vice-versa. Validated by DualEndpointTests in tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ — boots two OpcUaApplicationHost instances on loopback, asserts a real OPCFoundation client Session reading Server.ServerArray from Node A sees both URIs.

Split-brain / downing

The default downing strategy is auto-down (changed 2026-07-21). It is selected by Cluster:SplitBrainResolverStrategy, and ServiceCollectionExtensions.BuildDowningHocon turns that into the akka.cluster.downing-provider-class actually installed:

`Cluster:SplitBrainResolverStrategy`	Provider installed	Posture
`auto-down` (default)	`Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s`	Availability. The leader among the reachable members downs the unreachable peer, so a crash of either node — oldest included — fails over in place.
`keep-oldest`	`Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf`	Partition-safety. A partition can never run dual-active. Not safe for a 2-node pair — see below.

Any other value fails the host at startup rather than falling through to a default.

Why keep-oldest is no longer the default

The previous documentation in this section — and the code comments behind it — asserted that keep-oldest with down-if-alone = on was "the correct strategy for a 2-node warm-redundancy pair." That was wrong, and the error was not conservative: it described a total-outage configuration as the recommended one.

In Akka.NET 1.5.62's KeepOldest.OldestDecision, the down-if-alone rescue branch requires the surviving side to hold ≥ 2 members. In a two-node cluster a lost peer is always a 1-vs-1 split, so the branch never fires and the lone survivor falls through to DownReachable — downing itself — and run-coordinated-shutdown-when-down = on then terminates it. down-if-alone is a 3+-node feature; it does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members. See ../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md for the upstream decision record, which OtOpcUa follows.

The alternatives do not help a pair: static-quorum with quorum 1 hits Akka's IsTooManyMembers guard and returns DownAll; with quorum 2 the survivor downs itself; keep-majority keeps the lowest-address side, which just moves the fatal crash from "the oldest node" to "the other node"; lease-majority needs a shared lease store both nodes can reach.

The accepted trade

Under auto-down, a genuine network partition (both nodes alive, link cut) leaves each side downing the other and continuing alone: both run active, both hold singletons, and both advertise the primary ServiceLevel. Recovery is operator-driven — after the link heals, restart one side; it rejoins as a fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual downing quarantines the association.

This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and a crash under keep-oldest was unrecoverable without operator action anyway.

Deployments that would rather take an outage than ever run dual-active can set Cluster:SplitBrainResolverStrategy: "keep-oldest" — but should read the section above first, because for a two-node pair that choice means any crash of the oldest node is a full outage.

Recovery still depends on supervision

A service supervisor that restarts the process on exit — production Install-Services.ps1 sets sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000; the docker-dev rig sets restart: unless-stopped.
The recovery watchdog ActorSystemTerminationWatchdog (registered in Program.cs after AddAkka) — it watches ActorSystem.WhenTerminated and, on an unexpected self-down, calls IHostApplicationLifetime.StopApplication() so the process exits and the supervisor restarts it instead of idling forever with a dead actor system. Under auto-down the survivor no longer needs this (it is never the one downed); the downed node still does.
Both peers listed in SeedNodes on every node, so a restarted node can re-join via either peer — plus the self-form fallback below, without which a node cold-starting while its peer is dead waits in InitJoin forever (auto-down removes the crash outage, not that one).

On HardKillFailoverTests: that in-process test hard-kills the oldest and asserts the survivor becomes sole driver role-leader. It passed even under keep-oldest — because it simulates the crash with provider.Transport.Shutdown(), which leaves node A's ActorSystem alive (a transport partition with the oldest still running), not a real process death. A real 2-container docker kill of the oldest downed the survivor (verified 2026-07-15, archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md). It is a standing example of a green test over a fatal defect: keep the distinction between a transport partition and a process death in mind when reading it.

SplitBrainResolverActivationTests pins the strategy by starting a real host through the production bootstrap and reading akka.cluster.downing-provider-class back off the running ActorSystem — asserting the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the losing one still looks correct in the file.

There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do automatically.

Live gate outstanding. The auto-down change is verified by unit tests against the effective configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has not yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster; assert the survivor stays up, takes the singletons, and reports the primary ServiceLevel) should run alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See docs/plans/2026-07-21-per-cluster-mesh-design.md §6.2 / Phase 0a.

Bootstrap: the self-form fallback

Listing both peers in SeedNodes does not mean either node can cold-start alone. Akka lets only the first listed seed form a new cluster; every other node sends InitJoin to the seeds and retries — forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up, and no downing strategy helps: auto-down removes the crash outage, not this one.

Cluster:SelfFormAfter closes that gap. ClusterBootstrapFallback.Arm (armed for every host built through WithOtOpcUaClusterBootstrap, via an Akka.Hosting AddStartup task) waits that long for cluster membership and, on expiry, calls Cluster.Join(SelfAddress) — the node forms a cluster on itself and becomes operational unattended.

`Cluster:SelfFormAfter`	Behaviour
`"00:00:10"` (default)	A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires.
`null` / `≤ 0`	Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely.

The island guard is the load-bearing part. The fallback fires only when this node's own address appears in its own SeedNodes. A node that is not one of its own seeds is never legitimately first, and if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is exactly the current docker-dev topology: the site-a/site-b driver nodes list only central-1 as a seed, so they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh (docs/plans/2026-07-21-per-cluster-mesh-design.md) each pair node is a seed of its own two-node mesh, so the fallback covers both sides of every pair and the site-node exception disappears.

Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a booting node only self-forms when no seed answered), so a peer booting after the survivor self-formed simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the window while mutually unreachable — both self-form, which is the same dual-active class the auto-down strategy already accepts, with the same recovery (restart one side).

Pinned by SelfFormBootstrapTests (tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/), which starts real hosts through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node absent from its own seed list never self-forms. The two negative cases each carry a positive control (an explicit self-join) so they cannot pass merely because the node was unformable.

Live gate: PASSED (docker-dev, 2026-07-22). Drilled on the six-node rig, defect first and fix second. On the pre-change image, central-2 started alone (central-1 stopped) and logged zero join events in 70 s — the InitJoin loop, observed rather than argued. On the rebuilt image the same node logged the self-form warning 10 s after start and Leader is moving node [central-2] to [Up], then came up with all four admin singletons. A site-a-2 recreated while central-1 was still down logged "self-form fallback inactive: this node … is not in its own seed list" and stayed out — the island guard on the real topology. Restarting central-1 produced Welcome from [central-2]: it joined the self-formed cluster instead of starting a second one, and the four site nodes then joined through it — six members, one cluster.

Primary data-plane gate (writes, acks, alerts emit)

Three data-plane surfaces are Primary-gated so a warm standby never touches a shared field device or duplicates a fleet-wide publish: inbound operator writes (DriverHostActor.HandleRouteNodeWrite), native-alarm acks (HandleRouteNativeAlarmAck), and the native/scripted alerts emit (ForwardNativeAlarm / ScriptedAlarmHostActor.OnEngineEmission). All three route through one policy — PrimaryGatePolicy.ShouldServiceAsPrimary(localRole, driverMemberCount) (OtOpcUa.Runtime.Drivers):

Local role (last redundancy snapshot)	Cluster driver members	Decision
`Primary`	any	service
`Secondary` / `Detached`	any	deny
unknown (no snapshot yet, or snapshot omitted this node)	≤ 1 (single-node / boot on a lone driver)	service (boot-window / single-node posture)
unknown	≥ 2 (a real driver peer exists)	deny (default-deny — a Primary peer exists; don't act until a snapshot proves this node is it)

Before archreview 03/S4 the unknown-role case defaulted to allow unconditionally, opening a dual-primary window: a freshly-booted secondary on a multi-node cluster serviced shared-device writes and emitted duplicate fleet-wide alerts for up to the ~10 s RedundancyStateActor heartbeat (and indefinitely if the snapshot's NodeId never matched this node). Membership is authoritative much earlier than DPS delivery, so an unknown role on a multi-driver cluster now default-denies. The driver member count is read live from Cluster.State.Members (Up members with the driver role); a non-cluster ActorRefProvider yields 0 ⇒ single-node posture. If a snapshot arrives that never mentions this node while a driver peer exists (the 03/S5 identity-mismatch shape), DriverHostActor logs a one-time Warning so the silent mismatch is diagnosable.

What a client sees during the boot window (multi-node, role unknown): an inbound write is rejected (not queued — OT actuation seconds late is a surprise). The reject rides the sanctioned optimistic-write self-correction surface: the node reverts to its prior value with a transient Bad-quality blip and a Part 8 AuditWriteUpdateEvent; the client's synchronous write call still returned Good (exactly as any failed device write behaves since the #5 write-outcome self-correction). The rejection reason distinguishes the boot window ("not primary (role unknown)") from a steady-state Secondary ("not primary"). Acks are dropped (Warning-logged when role-unknown). The alerts emit is skipped (the Primary peer publishes the single fleet copy). The window closes on the first delivered snapshot.

Under warm/hot redundancy both cluster nodes run ScriptedAlarmHostActor and evaluate scripted alarms, keeping each node's address space and engine state warm for instant failover. To avoid duplicate rows on /alerts and duplicate historian writes, only the Primary node publishes externally:

alerts topic emission — ScriptedAlarmHostActor and DriverHostActor.ForwardNativeAlarm subscribe to the redundancy-state DPS topic and cache the local node's RedundancyRole, then gate the cluster alerts publish through PrimaryGatePolicy (table above). The OPC UA condition-node write and inbound ack/shelve command processing remain ungated on both nodes so the secondary is always ready to serve clients after a failover.
HistorianAdapterActor historization — likewise Primary-gated so alarm historization is exactly-once across all alarm sources. The actor subscribes to the alerts DPS topic and translates each AlarmTransitionEvent → AlarmHistorianEvent before enqueuing it on the sink; scripted alarms therefore historize exactly once regardless of cluster size.
The alarm sink's DRAIN — a second, independent gate on the same decision. The enqueue gate above is not sufficient any more: since Phase 2 the store-and-forward buffer lives in the replicated LocalDb, so the Secondary holds a full copy of the Primary's queued rows and an ungated drain there would re-deliver every event continuously. LocalDbStoreAndForwardSink takes a drainGate fed from IRedundancyRoleView — a singleton DriverHostActor publishes its PrimaryGatePolicy verdict to, because the drain runs on a timer and cannot receive RedundancyStateChanged itself. A gated-off node reports HistorianDrainState.NotPrimary. Delivery is at-least-once across a failover by design: rows the old Primary delivered just before dying may not have replicated their deletes yet. See AlarmHistorian.md.

Net effect: each alarm transition appears once on /alerts and would historize once, not once per node.

See ScriptedAlarms.md and AlarmTracking.md for the scripted-alarm engine internals.

Pair-local store (LocalDb — Phases 1 + 2)

Independently of the ServiceLevel machinery above, every driver-role node keeps a consolidated ZB.MOM.WW.LocalDb SQLite database. Phase 2 added the alarm store-and-forward buffer (alarm_sf_events) to it alongside the config cache, so a node that dies holding undelivered alarm history no longer takes it to the grave — see the Primary-only drain note above and AlarmHistorian.md. Phase 1 caches the deployed-configuration artifact (chunked). Its job is a single failure mode redundancy does not otherwise cover: a driver node restarting into a central-SQL-Server outage would come up with no configuration at all. With the cache, it boots from the last artifact it applied instead, and logs a running-from-cache signal.

What replicates. The cache can optionally replicate to the node's redundant pair peer over the LocalDb library's gRPC sync (HLC-stamped, last-writer-wins). Two tables replicate: deployment_artifacts and deployment_pointer. Replication is default-OFF and fail-closed — it stays inert until LocalDb:SyncListenPort + a peer are configured, and a missing/mismatched LocalDb:Replication:ApiKey refuses or silently halts sync. It is enabled per-pair as an opt-in.
The replicated-cache payoff. In a pair with replication on, either node can be the one that applied the latest deploy; the peer holds a byte-identical copy. So a node that restarts during a central outage can boot the current config even if it never applied that deploy itself — its peer did, and the row replicated.
What it does NOT cover. The cache is a boot fallback for central-SQL outages only, not a standby data path and not a replacement for central. A new deployment still requires central SQL; the cache is read only when the central fetch fails at startup, and the node resumes fresh-config behavior once central returns. It does not replicate live tag values, alarms, or historian data — only the config artifact. It is orthogonal to the ServiceLevel/primary-gate logic: a node running from cache still participates in redundancy normally.

Full detail: the enablement runbook at docs/operations/2026-07-20-localdb-pair-replication.md and the LocalDb section of docs/Configuration.md.

Client-side failover

The OtOpcUa Client CLI at src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI supports -F / --failover-urls for automatic client-side failover; for long-running subscriptions the CLI monitors session KeepAlive and reconnects to the next available server, recreating the subscription on the new endpoint. See Client.CLI.md.

Observability

OpcUaPublishActor emits one metric on every ServiceLevel transition (it suppresses no-op repeats of the same byte):

Metric	Type	Notes
`otopcua.redundancy.service_level_change`	Counter (`{change}`)	OPC UA `Server.ServiceLevel` transitions emitted by the redundancy state. Tagged with `level` = the new byte.
`otopcua.redundancy.primary_gate_denied`	Counter (`{denial}`)	Operations denied by the Primary data-plane gate (archreview 03/S4). Tagged `site` = `write` \| `ack` \| `alarm-emit` and `reason` = `secondary` \| `detached` \| `role-unknown`. A non-zero `role-unknown` rate on a healthy multi-node cluster flags a stuck boot window (redundancy snapshot not delivering).
`otopcua.opcua.apply.failed`	Counter (`{apply}`)	Address-space apply/materialise passes that swallowed at least one sink failure (archreview 01/S-1). Tagged `kind` = `rebuild` (the rebuild threw) \| `nodes` (per-node materialise failures). A non-zero rate means the running server holds a stale or partial address space despite a reported-successful deploy — `OpcUaPublishActor` also logs the degraded apply at Error.
`galaxy.writes.advise_failed`	Counter (`{write}`)	Galaxy writes short-circuited to `Bad` because `AdviseSupervisory` failed — the value could not have committed, so the write fails closed (archreview 06/S-1) and the Bad status fires the #5 node revert instead of leaving a phantom-Good node. On the Galaxy driver meter `ZB.MOM.WW.OtOpcUa.Driver.Galaxy`.
`galaxy.writes.unconfirmed`	Counter (`{write}`)	Galaxy writes reported `Good` off an empty gateway statuses array (command accepted; the COM-side commit is unconfirmed pending the mxaccessgw `WriteComplete` correlation follow-up). The honest signal for the optimistic-Good rate until that cross-repo fix lands. On the Galaxy driver meter.

The redundancy/apply meters are defined on OtOpcUaTelemetry (src/Core/ZB.MOM.WW.OtOpcUa.Commons/Observability/OtOpcUaTelemetry.cs); the Galaxy write meters hang off the driver's own meter (EventPump.MeterName). All surface through whatever OpenTelemetry exporter the host configures.

Galaxy fail-closed write semantics (archreview 06/S-1): a non-secured Galaxy write commits only when its item is AdviseSupervisory-advised first (the writer runs with no login). If the advise fails, the write is not issued — GatewayGalaxyDataWriter returns BadCommunicationError (a knowingly-lost write is never reported as Good). An empty statuses array on the write reply is still treated as Good but metered as galaxy.writes.unconfirmed. See Historian.md / the Galaxy driver notes for the write path.

Depth reference

For the full design — message contracts, tiered calculator truth table, recovery semantics — see docs/plans/2026-05-26-akka-hosting-alignment-design.md §6.

36 KiB Raw Blame History Unescape Escape