scadaproj/akka_failover.md

Akka.NET HA, Failover & Singletons — ScadaBridge + OtOpcUa Deep Dive

Written 2026-07-22 against ScadaBridge main @ 69b3ccfc and OtOpcUa (lmxopcua) master @ 0de1352e. Both trees contain the 2026-07-21 downing/election corrections (ScadaBridge cf3bd52f; OtOpcUa 2964361a auto-down + 50b55ee4 oldest-driver election — note 0de1352e itself is the adjacent NU1903 dependency pin, not the redundancy fix).

Design priority (stated 2026-07-22): for the 2-node site pairs, a network-partition split-brain is less risky than failing to bring the second node up quickly. HA goal: either node can be launched first, automatic failover, no manual intervention. The architecture below is evaluated against that priority throughout.

---

1. How Akka.NET handles failover — the fundamentals both apps build on

The deployment unit at every site is a 2-node Akka.NET cluster (one node per Windows VM). Central is also a 2-node cluster, with SQL Server; sites have no SQL Server. Everything HA-related decomposes into five Akka mechanisms:

1.1 Membership and seed nodes

A cluster forms by gossip. Each node lists seed-nodes; a booting node sends InitJoin to the seeds. Only the FIRST seed in the list may join itself to form a new cluster — any other node waits until some existing member answers. Both apps set min-nr-of-members = 1, so a single node (if it's allowed to form) goes Up alone and hosts all singletons — the pair does not need both members to become operational.

1.2 Failure detection (the heartbeat)

Every node heartbeats its peers and feeds the intervals into a Phi Accrual failure detector. When the suspicion level crosses the threshold, the peer is marked Unreachable — a suspicion, not a removal. Both apps run the same values:

Setting	Value (both apps)	Meaning
failure-detector.heartbeat-interval	2 s	Cluster-membership heartbeat rate
failure-detector.threshold	10.0	Phi threshold (suspicion sensitivity)
failure-detector.acceptable-heartbeat-pause	10 s	Tolerated silence before Unreachable
transport-failure-detector (remoting layer)	2 s / 10 s pause	Detects dead TCP associations

So a hard-killed peer is declared Unreachable roughly 10–12 s after death. These are exactly the "heartbeat with a small adjustable timeout" — in ScadaBridge they are bound from config (Cluster:HeartbeatInterval, Cluster:FailureDetectionThreshold → ClusterOptions, rendered into HOCON by AkkaHostedService.BuildHocon); in OtOpcUa they live in akka.conf (lines 68–72) in the Cluster library.

1.3 Downing (the decision to act)

Unreachable alone changes nothing — the leader cannot make membership decisions while any member is unreachable, singletons do not move, the cluster is wedged. Something must Down the unreachable member. This is the strategy choice, and it is the whole availability-vs-partition-safety trade:

• Auto-downing (Akka.Cluster.AutoDowning + auto-down-unreachable-after): after the window, the leader among the reachable members downs the unreachable one. In a 1-vs-1 split, each side downs the other — the crash case fails over perfectly; a genuine partition produces dual-active.
• Split Brain Resolver keep-oldest (+ down-if-alone): keeps the side containing the oldest member. Partition-safe, but fatally broken for a 2-node pair: in Akka.NET 1.5.62, KeepOldest.OldestDecision only lets down-if-alone rescue a side holding ≥ 2 members (otherSide == 1 && thisSide >= 2). A 1-vs-1 survivor whose peer (the oldest) crashed takes DownReachable and downs itself — proven live: SBR took decision Akka.Cluster.SBR.DownReachable … including myself, [1] unreachable of [2] members. Net effect: the pair could not survive a crash of the oldest node — total outage.

Both apps switched their default from keep-oldest to auto-down on 2026-07-21 (ScadaBridge cf3bd52f, decision record ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md; OtOpcUa 2964361a, same fix ported). The owner rationale recorded in the decision doc is verbatim the priority stated above: "network partitions are less of a risk than if this stops working."

Rejected alternatives (from the decision record): static-quorum(1) → DownAll (worse); static-quorum(2) → survivor self-downs; keep-majority → keeps the lowest-address side, same fatal crash re-keyed; lease-majority → needs a shared lease store (no SQL/K8s at sites); a third arbiter node → no third VM at sites; a custom downing provider → would reimplement AutoDowning.

1.4 Cluster singletons (what actually "fails over")

ClusterSingletonManager runs on every node of the (optionally role-filtered) cluster and guarantees exactly one instance of the actor, hosted on the oldest member of that role. ClusterSingletonProxy routes to wherever it currently lives. Failover semantics:

• Graceful shutdown (service stop, redeploy): hand-over protocol — the old host transfers to the new oldest with no dead window beyond hand-over latency. Both apps hook CoordinatedShutdown (run-by-clr-shutdown-hook = on; ScadaBridge additionally drains each singleton via a PhaseClusterLeave GracefulStop task, 10 s per singleton, phases.cluster-leave.timeout = 15s).
• Hard crash: the singleton is simply gone until the dead node is Downed; then the new oldest member's manager instantiates a fresh instance. This is why downing is the linchpin — no downing decision, no singleton recovery, ever.

1.5 Oldest-member semantics — the family-wide rule

Both codebases converged on the same insight: Akka's "leader" (lowest address) and "oldest member" (singleton placement) diverge permanently after any node restart — the restarted node rejoins as a new incarnation, becoming the youngest, but may still have the lowest address and thus be leader. Any "active node" decision keyed on leader/lowest-address can therefore point at a node that is not hosting the singletons. Both apps now derive "active/primary" strictly from oldest Up member (ScadaBridge ActiveNodeEvaluator.SelfIsOldestUp; OtOpcUa RedundancyStateActor.SelectDriverPrimary with Member.AgeOrdering), which by construction matches singleton placement.

---

2. The failover timeline (and the knobs that tune it)

End-to-end failover after a hard node death, with current production values:

t=0        node dies (process crash / VM loss)
t≈10-12s   failure detector: peer marked Unreachable        ← acceptable-heartbeat-pause (10s)
t≈25-27s   auto-down window expires; survivor downs peer    ← auto-down-unreachable-after (15s)
t≈25-30s   survivor is now oldest; singletons instantiate;
           primary/active gates flip; ServiceLevel 250 moves

Measured, not theoretical: ScadaBridge's 2026-07-21 docker drill measured 28 s (active-node crash → standby takeover) and 27 s (standby crash → removal, zero routing blips); the in-process FailoverTimingTests measured 33.7 s for a full cycle.

Tuning for faster failover (per the availability-first priority):

Knob	Where	Current	Effect of lowering
acceptable-heartbeat-pause	SB: Cluster:FailureDetectionThreshold; OtOpcUa: akka.conf	10 s	Faster Unreachable, but < ~5 s risks false positives from GC pauses / Windows VM scheduling stalls / snapshot freezes
heartbeat-interval	SB: Cluster:HeartbeatInterval; OtOpcUa: akka.conf	2 s	More samples → detector converges faster; cheap to lower to 1 s
Auto-down window	SB: Cluster:StableAfter; OtOpcUa: DowningStableAfter const (15 s) + emitted HOCON	15 s	Direct 1:1 reduction of failover time. Every second cut is a second of dual-active risk during a transient blip (link flap, VM pause) — the window exists to let transients heal before the irreversible Down
down-removal-margin (OtOpcUa akka.conf)	15 s	Delay before rebalancing after removal	Keep ≈ downing window

Constraint pinned by an OtOpcUa test (Downing_window_exceeds_the_acceptable_heartbeat_pause): the downing window must exceed acceptable-heartbeat-pause, otherwise a node can be downed on a pause the detector was configured to tolerate. A realistic aggressive profile for the site pairs would be ~5 s pause + ~7–8 s window ≈ 12–13 s total failover, at the cost of downing a peer that stalls (not dies) for > ~12 s — acceptable per the stated priority, but worth validating against real Windows VM GC/backup/snapshot pauses before rollout. In ScadaBridge these are plain per-deployment config values; in OtOpcUa the 15 s window is currently a constant in ServiceCollectionExtensions.cs (only the strategy is config-selectable), so making the window configurable would be a small change there.

What happens to the dead node's side: both apps set run-coordinated-shutdown-when-down = on — a node that gets Downed (e.g. it was merely partitioned and the other side downed it) terminates its own ActorSystem. Both apps run a termination watchdog (ScadaBridge in AkkaHostedService; OtOpcUa ActorSystemTerminationWatchdog) that distinguishes intentional stop from self-down and calls StopApplication(), so the process exits and the service supervisor restarts it (Windows: sc.exe failure … restart/5000/restart/30000/restart/60000; Docker: restart: unless-stopped). The restarted process rejoins as a fresh incarnation — youngest member, i.e. it comes back as the standby, never usurping the survivor.

---

3. ScadaBridge

3.1 Cluster formation

All HOCON is generated in code — AkkaHostedService.BuildHocon(...) (src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs:258–329) from ClusterOptions/NodeOptions (src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs). Key facts:

• Seed nodes: central docker pair → scadabridge-central-a:8081 then …-central-b:8081 (both nodes list central-a first). Site pairs analogous on 8082.
• MinNrOfMembers = 1 (validator requires exactly 1) — one node forms the cluster and brings up all singletons.
• Roles: central node = ["Central"]; site node = ["Site", "site-{SiteId}"] (BuildRoles, line 406). The role drives the entire composition branch in Program.cs.
• Ports: central remoting 8081; site remoting 8082, gRPC 8083, metrics 8084.

3.2 Downing configuration

Configurable via Cluster:SplitBrainResolverStrategy, allowed values auto-down | keep-oldest (case-insensitive; validator ClusterOptionsValidator). Default and all 16 appsettings files: auto-down. The HOCON branch (AkkaHostedService.cs:275–286):

var downingBlock = string.Equals(strategy, "auto-down", OrdinalIgnoreCase)
    ? $@"downing-provider-class = ""Akka.Cluster.AutoDowning, Akka.Cluster""
        auto-down-unreachable-after = {DurationHocon(clusterOptions.StableAfter)}"
    : /* SBR provider + split-brain-resolver { active-strategy, stable-after,
         keep-oldest { down-if-alone } } */;

StableAfter = 00:00:15 doubles as the SBR stable-after and the auto-down window. DownIfAlone is now validated only under keep-oldest (inert under auto-down). keep-oldest remains supported and SBR-path tests still pin it (SbrFailoverTests.HardCrashOfYoungerNode_SbrDownsIt_AndOldestKeepsSingleton).

3.3 Cluster singletons

All registered through SingletonRegistrar.Start(...) (names {name}-singleton / {name}-proxy, PoisonPill termination, optional role, per-singleton PhaseClusterLeave drain task of 10 s).

Central — 7 singletons (role not further scoped; central nodes are all Central):

Singleton	Actor	Purpose
notification-outbox	NotificationOutboxActor	Notification ingest/dispatch/purge
audit-log-ingest	AuditLogIngestActor	Central audit ingest from sites
site-call-audit	SiteCallAuditActor	SiteCalls upserts + central→site retry/discard relay
audit-log-purge	AuditLogPurgeActor	Daily partition-switch purge
site-audit-reconciliation	SiteAuditReconciliationActor	Per-site audit reconciliation pull
kpi-history-recorder	KpiHistoryRecorderActor	KPI sampling (not readiness-gated)
pending-deployment-purge	PendingDeploymentPurgeActor	Expired staging-row reclaim (not readiness-gated)

RequiredSingletonsHealthCheck probes the first five via bounded Identify for /health/ready; the last two are deliberately best-effort.

Site — 2 singletons, role-scoped to site-{SiteId} so each site pair elects its own:

Singleton	Actor	Purpose
deployment-manager	DeploymentManagerActor	Deployment lifecycle; drains in-flight SQLite writes on hand-over
event-log-handler	EventLogHandlerActor	Event-log queries always served by the active node (event log is node-local, not replicated)

Deliberately not singletons: dcl-manager (DataConnectionManagerActor) and CertStoreActor run on every site node — the deployment singleton fans PKI trust changes to both nodes' cert stores so a failover never finds a stale trust store.

3.4 Active-node gating

One definition of "active": ActiveNodeEvaluator.SelfIsOldestUp (src/ZB.MOM.WW.ScadaBridge.Communication/ClusterState/ActiveNodeEvaluator.cs) — oldest Up member, explicitly not cluster.State.Leader (the XML docs spell out the leader-vs-oldest divergence after restarts). Consumers:

• Central inbound API: IActiveNodeGate → InboundApiEndpointFilter returns HTTP 503 {code:"STANDBY_NODE"} on the standby; Traefik routes on /health/active (OldestNodeActiveHealthCheck + DatabaseHealthCheck), so the proxy and the API agree on which node is active.
• Site store-and-forward: storeAndForwardService.SetDeliveryGate(() => clusterNodeProvider.SelfIsPrimary) — only the active node runs the delivery sweep; without the gate both nodes would deliver the same replicated sf_messages rows (systematic duplicate external calls). Re-evaluated per sweep tick, so delivery resumes within one RetryTimerInterval after failover.
• Heartbeat to central: SiteCommunicationActor stamps HeartbeatMessage.IsActive from the same evaluator (exception-safe, defaults to false).

Takeover is purely membership-driven: peer Downed → survivor becomes oldest → SelfIsOldestUp flips → singletons, S&F gate, /health/active, heartbeat all follow.

3.5 Site state without SQL Server — LocalDb

Sites persist everything in one consolidated SQLite file (LocalDb:Path, required, on the data volume) via ZB.MOM.WW.LocalDb — 10 replicated tables: OperationTracking, site_events, sf_messages (the store-and-forward buffer), and 7 config tables (deployed configurations, static overrides, shared scripts, external systems, DB connections, DCL definitions, native alarm state).

Pair replication (default OFF; opt-in LocalDb:Replication:PeerAddress + matching ApiKey both sides; fail-closed auth interceptor; rides the existing gRPC 8083 listener): CDC triggers capture row mutations, shipped bidirectionally under HLC last-writer-wins; snapshot resync merges per row and never deletes. This is what makes site failover real: the standby already holds the S&F buffer and the full config, so on takeover it just opens its delivery gate over data it already has — zero cross-node fetch (the old notify-and-fetch deploy path is deleted).

Not replicated (by design): the event log (hence the event-log-handler singleton), in-memory alarm state (re-evaluated from incoming values on the survivor), and notification_lists/smtp_configurations (would ship plaintext SMTP passwords).

Operational constraints:

• Stop/start a site pair TOGETHER. Rolling upgrades are unsupported: Phase 2 deleted the legacy replicator and its SfBufferSnapshot compat handler, so a mixed-version pair runs but silently does not converge.
• A node offline longer than TombstoneRetention (7 d) can resurrect deleted rows on rejoin; within the window rejoin is safe and self-correcting.
• MaxBatchSize counts rows, not bytes — rig pins 16 (the 500 default could build a ~35 MB batch against the 4 MB gRPC cap).

3.6 Verified behavior + remaining gaps

• Auto-down failover drill (docker failover-drill.sh, SIGKILL): active-crash takeover 28 s, standby-crash 27 s, 0 routing blips; both drill modes now assert recovery.
• After a real partition the two mutually-downed halves do not merge (quarantined association) — an operator restarts one side, which rejoins as standby. This is the accepted dual-active trade.
• The old "restart central-a and central-b together (SBR self-down)" gotcha is closed by auto-down — but a related constraint survives with a different mechanism: the seed-node bootstrap constraint (§6.1).
• ⚠ Stale doc: docs/deployment/topology-guide.md:101 still says keep-oldest + down-if-alone. Authoritative: Component-ClusterInfrastructure.md, the auto-down decision record, docker/README.md.

---

4. OtOpcUa

4.1 Cluster formation

Base HOCON in the library (src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf): system otopcua, remoting port 4053, min-nr-of-members = 1, empty seeds/roles overlaid at runtime from AkkaClusterOptions (Cluster:* config; roles restricted to admin | driver | dev by RoleParser). Bootstrap via Akka.Hosting: WithOtOpcUaClusterBootstrap (remoting + clustering + the downing HOCON, Prepended). Note the dual role plumbing: OTOPCUA_ROLES drives DI composition (hasAdmin/hasDriver branches in Program.cs), while Cluster__Roles__* drives the actual Akka member roles — docker-dev sets both (a past production incident came from setting only OTOPCUA_ROLES, leaving the Akka member role-less).

Current mesh shape (important): today OtOpcUa runs one Akka mesh for the whole fleet — docker-dev is six nodes (central-1/2 = admin,driver; site-a-1/2, site-b-1/2 = driver) all seeded by central-1, with logical MAIN/SITE-A/SITE-B separation done by ServerCluster.ClusterId rows in the shared ConfigDb, not by separate meshes. The per-cluster mesh design (§4.4) will change this to one mesh per 2-node pair — the shape that matches your site deployment.

4.2 Downing configuration

Same key as ScadaBridge: Cluster:SplitBrainResolverStrategy, default auto-down, allowed auto-down | keep-oldest, anything else fails the host at startup (ArgumentOutOfRangeException — no silent fallback; pinned by Unknown_strategy_throws_rather_than_silently_falling_back). BuildDowningHocon emits:

akka.cluster {
    downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
    auto-down-unreachable-after = 15000ms
}

(the comment notes auto-down-unreachable-after defaults to off upstream and is load-bearing — AutoDowning without it never downs anything). Under keep-oldest the typed KeepOldestOption { DownIfAlone = true } installs the SBR instead; the split-brain-resolver block in akka.conf is inert under the default. Tests read the effective config off a running ActorSystem (not the input HOCON) — provider class, 15 s window, case-insensitivity, and the window-exceeds-heartbeat-pause invariant.

Live gate still open: the 1-vs-1 oldest-crash pathology cannot be exercised on the current six-node docker-dev mesh; the crash-the-oldest drill is deferred to the per-cluster mesh work (Phase 6/7). Supporting live evidence exists from a 2-container kill test (archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md) and from ScadaBridge's identical fix. Cautionary note recorded in docs/Redundancy.md: the pre-existing HardKillFailoverTests stayed green under broken keep-oldest because it shut down the transport (ActorSystem alive) rather than killing the process — "a standing example of a green test over a fatal defect."

4.3 Redundancy primary election + ServiceLevel

• RedundancyStateActor (an admin-role singleton) computes the fleet snapshot: SelectDriverPrimary = oldest Up member carrying the driver role (Member.AgeOrdering; Leaving excluded — a leaving node is handing its singletons over and must not be Primary). Debounced 250 ms, republished on a 10 s heartbeat over distributed pub/sub topic redundancy-state (DPS doesn't replay to late subscribers).
• NodeRedundancyState carries RedundancyRole { Primary, Secondary, Detached } and IsDriverPrimary (renamed from IsRoleLeaderForDriver — "the name now describes what the value means rather than how it used to be computed").
• ServiceLevel (ServiceLevelCalculator): health basis 240 (healthy) / 200 (stale) / 100 (critically degraded) / 0 (down), +10 if driver Primary, so a healthy pair reads 250 / 240. Published into the OPC UA Server.ServiceLevel variable via SdkServiceLevelPublisher (deferred publisher until the SDK server exists; first computed value always published so no node lingers at the SDK default 255).
• Client-visible failover is OPC UA non-transparent redundancy: each node has its own ApplicationUri; clients read Server.ServerArray (self + peers, from PeerApplicationUris) and pick the endpoint with the highest ServiceLevel. The shipped client (OpcUaClientService) does this: failover URL list, KeepAlive-failure trigger, single-flight failover guard, data + alarm subscription replay after reconnect. Consumer note: because Primary = oldest (not lowest address), after any node restart a ServiceLevel-selecting client may legitimately prefer a different node than before — that's correct behavior, not a bug.

Primary gating of the data plane (PrimaryGatePolicy) — the S4 default-deny fix:

localRole switch {
    Primary                => true,
    Secondary or Detached  => false,
    _ /* unknown */        => driverMemberCount <= 1,  // multi-driver: DENY until proven
};

During the boot window (role not yet known) a multi-driver node rejects device writes with not primary (role unknown) (denial meter reason=role-unknown); a single-driver node stays default-allow. Gated surfaces: device writes, native-alarm acks, alarm-alert fan-out. A deliberately different policy governs alarm-history drain: unknown → drain anyway ("a duplicate row beats a silent gap"), keyed on whether the Primary shares this node's queue (pair-local). Live-proven on a 2-node docker rig (R2-04 T15): ServiceLevel 250/240, boot-window write rejected, steady-state secondary rejected + reverted, primary write reaches the device.

Failover sequence when the primary dies: peer downed (auto-down) → survivor is oldest driver → SelectDriverPrimary flips → snapshot republished → gates open on the survivor → ServiceLevel 250 moves → OPC UA clients see the old endpoint die (KeepAlive) and re-select by ServiceLevel. Singletons migrate to the same node the election names, because both derive from member age.

4.4 Singletons and the per-cluster mesh gap

Five control-plane singletons, all role-pinned to admin:

Singleton	Actor	Purpose
config-publish	ConfigPublishCoordinator	Persists per-node deploy ACKs
admin-operations	AdminOperationsActor	Operator ack/shelve; TagConfig deploy gate
audit-writer	AuditWriterActor	Audit persistence
fleet-status	FleetStatusBroadcaster	AdminUI fleet panel
redundancy-state	RedundancyStateActor	Primary election + DPS publisher

KNOWN LIMITATION: the Primary election is scoped per Akka cluster, not per application Cluster. On the one-mesh fleet, SelectDriverPrimary names exactly ONE Primary among all driver nodes (MAIN + SITE-A + SITE-B), while every gated resource is pair-local. The per-cluster mesh design (OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md) fixes this by construction: one Akka mesh per application Cluster (two nodes max), central↔cluster joined by explicit transports (modeled on ScadaBridge's ClusterClient/gRPC/HTTP trio), so "am I the driver Primary?" and "the resource I'm gating" have the same scope. Status: Phases 0a (auto-down) and 0b (oldest-Up election) DONE and merged; Phases 1–7 (ClusterNode columns, comm actors, config fetch-and-cache, cut driver-side ConfigDb, gRPC stream contract, mesh partition, failover drill/live gate) not started — but now sequenced by a program plan (OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md, 2026-07-22), driven by the decision that OtOpcUa pairs will run on the SAME Windows VMs as the ScadaBridge nodes — two independent, identically-postured 2-node clusters per site, and driver nodes off the ConfigDb entirely (sites have no SQL Server).

4.5 Surviving central-SQL loss — availability guarantees

Two mechanisms make a driver pair independent of central SQL Server availability:

• Address-space availability guarantee (#485/#486): an artifact the node cannot read is no answer, never an empty configuration. Previously a transient ConfigDb error mid-deploy parsed zero bytes into an empty composition → the planner diffed it as a PureRemove of every node → served address space emptied, drivers stopped, subscriptions cleared, deploy still reported Applied. Now OpcUaPublishActor / DriverHostActor hold last-known-good address space, drivers, and subscriptions, and report the deploy Failed without advancing the revision so the retry lands.
• LocalDb Phase 1 boot-from-cache: every driver node caches the chunked, SHA-256 verified deployed-configuration artifact in a pair-local SQLite DB; a node restarting into a central-SQL outage boots from the last artifact it applied. With opt-in pair replication (deployment_artifacts + deployment_pointer, HLC last-writer-wins, fail-closed auth) either node holds a byte-identical copy even if it never applied that deploy itself. Scope: config artifact only — no live tags/alarms/historian data.

---

5. Dual-active: what it concretely means here

Under auto-down, a genuine partition (both nodes alive, link cut) ends with each side downing the other and running active until an operator restarts one side (the mutual downing quarantines the association — the sides will not self-merge). Accepted per the availability-first decision; concrete blast radius:

ScadaBridge site pair: both nodes' SelfIsOldestUp = true → both open the S&F delivery gate → duplicate external deliveries of the same replicated sf_messages rows for the partition's duration; both stamp IsActive heartbeats. LocalDb replication itself is partition-tolerant (HLC last-writer-wins reconverges on heal, after the operator restart). Central pair: both serve /health/active = 200 and run duplicate singleton sets against the same SQL Server — duplicated sweeps/purges (mostly idempotent upserts), and Traefik may alternate between two "active" backends.

OtOpcUa pair: both nodes elect themselves driver Primary → both publish ServiceLevel 250 → clients on either side keep their session; both sides accept device writes → two masters against the same field devices for the duration. This is the classic OT dual-primary risk — mitigated by rarity (2 VMs, one LAN segment, real partitions ≈ switch failure) and by the alternative (keep-oldest) having been proven to turn a plain crash into a total outage.

Detection/ops: the survivor logs the downing decision; OtOpcUa emits otopcua.redundancy.service_level_change and primary_gate_denied meters; two simultaneous 250s / two 200-responses on /health/active are the alarm signal. Recovery is always "restart one side; it rejoins as youngest/standby."

---

6. Gaps vs the stated HA goal ("either node first, automatic failover")

6.1 The seed-node bootstrap constraint — CLOSED 2026-07-22 (differently per repo)

Automatic failover: achieved. Either node crashing is survivable, unattended, in ~27–30 s. The remaining gap — a non-first seed cold-starting alone loops on InitJoin forever (the "registered outage gap") — was closed in both repos on 2026-07-22, but execution overturned the original design and the two repos legitimately diverged:

⚠️ The original design's safety claim was FALSE — proven independently in both repos. The planned SelfFormAfter watchdog assumed "window expires mid-join-handshake → benign, Akka ignores Join once joined." Wrong: the watchdog sits outside Akka's join handshake and cannot distinguish no seed answered from a seed answered and the join is in flight. On a routine restart into a live peer, the join stalls briefly behind removal of the node's own stale incarnation; a Join(self) fired in that window abandons the in-flight join and forms a second cluster at the same address — a permanent island. ScadaBridge proved it with a written test (split still unhealed after 90 s); OtOpcUa hit it live when its manual-failover drill bounced a node (InitJoinAck received, no Welcome inside the window, fallback fired → second cluster).

ScadaBridge's fix — self-first seed ordering (no runtime code). Every node lists ITSELF as seed-nodes[0], partner second. Akka's own FirstSeedNodeProcess then implements exactly the intended semantics inside the handshake (InitJoin the other seeds; self-join only if nobody answers) — race-free by construction. A StartupValidator rule hard-gates the ordering at boot. Live-proven behaviors: lone cold-start self-joins after ~5 s (seed-node-timeout); restart into a live peer rejoins (never islands); simultaneous reachable cold-start converges to ONE cluster (the old "two clusters that never merge" objection was disproved by test); a genuine boot partition dual-forms — the accepted auto-down trade. There is no SelfFormAfter option in ScadaBridge — the whole watchdog was rejected in code review.

OtOpcUa's fix — converged on self-first ordering the same day. OtOpcUa first shipped the watchdog and hit the race live (its manual-failover drill bounced a node; the fallback islanded it), patched it with a TCP reachability guard — and then retired the whole watchdog hours later (lmxopcua master @ 3f24d4d6, a breaking change): ClusterBootstrapFallback and Cluster:SelfFormAfter are deleted, docker-dev central-2 now lists itself as SeedNodes__0, and a new AkkaClusterOptionsValidator enforces the ordering at boot. Two subtleties its rule handles that ScadaBridge's didn't need: the invariant is conditional (self must be seed[0] only IF self appears in the list at all — site nodes seed off central only and are deliberately not seeds), and identity compares on PublicHostname + port (the address Akka puts in SelfAddress), because matching the 0.0.0.0 bind address would leave the rule silently inert on the whole rig. The retirement was proven by a falsifiability control: the old peer-first ordering test failed at 11 s against the watchdog — demonstrating the watchdog could never usefully fire for the nodes that needed it.

Result: both repos now share the identical mechanism — self-first seed ordering, validator-enforced, no bootstrap runtime code. The mesh program's Phase 6 convergence item is already done; what remains there is only the per-pair seed topology itself.

Ops action (outstanding): the gitignored ScadaBridge/deploy/wonder-app-vd03/ overlay must have its SeedNodes reordered self-first before its next deploy — the new validator rule is a deliberate hard boot gate.

6.2 Other open items

• OtOpcUa auto-down live gate still open (needs a real 1-vs-1 rig; deferred to per-cluster mesh Phase 6/7). ScadaBridge's identical mechanism is drill-proven.
• OtOpcUa election scope is cluster-wide until per-cluster mesh Phases 1–7 land (§4.4) — only relevant while multiple pairs share one mesh.
• OtOpcUa's 15 s downing window is a code constant, not config — worth making tunable alongside any heartbeat-timeout reduction (ScadaBridge's already is: Cluster:StableAfter).
• ScadaBridge site pairs: stop/start together for upgrades (no mixed-version LocalDb replication); ≥ 7-day-offline nodes risk tombstone resurrection.
• Stale doc: ScadaBridge/docs/deployment/topology-guide.md:101 still describes keep-oldest.

---

7. Quick reference — file map

Concern	ScadaBridge	OtOpcUa
HOCON / downing emit	Host/Actors/AkkaHostedService.cs:258–329	Cluster/ServiceCollectionExtensions.cs:119–146 + Cluster/Resources/akka.conf
Options + validation	ClusterInfrastructure/ClusterOptions.cs, ClusterOptionsValidator.cs	Cluster/AkkaClusterOptions.cs, RoleParser.cs
Strategy key	Cluster:SplitBrainResolverStrategy (default auto-down)	same key, same default; unknown value = startup failure
Active/primary definition	Communication/ClusterState/ActiveNodeEvaluator.cs (oldest Up)	ControlPlane/Redundancy/RedundancyStateActor.cs (oldest Up driver)
Data-plane gating	InboundApiEndpointFilter (503 STANDBY_NODE), S&F SetDeliveryGate	Runtime/Drivers/PrimaryGatePolicy.cs + DriverHostActor gates
Singleton registration	Host/Actors/SingletonRegistrar.cs (7 central + 2 site)	ControlPlane/ServiceCollectionExtensions.cs:44–94 (5, role admin)
Client-visible failover	Traefik on /health/active; gRPC A/B flip	OPC UA ServiceLevel 250/240 + Server.ServerArray; client KeepAlive failover
Self-down recovery	watchdog in AkkaHostedService → process exit → supervisor	Host/ActorSystemTerminationWatchdog.cs → same
Decision records	docs/plans/2026-07-21-auto-down-availability-decision.md, docs/requirements/Component-ClusterInfrastructure.md	docs/Redundancy.md §"Split-brain / downing", docs/plans/2026-07-21-per-cluster-mesh-design.md
Failover drills / tests	docker/failover-drill.sh (28 s/27 s), SbrFailoverTests, FailoverTimingTests	SplitBrainResolverActivationTests, RedundancyPrimaryElectionTests, R2-04 T13/T15 live gate