dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a61e041e58a0db47b3465c2b650f39124e70ee86
scadaproj/docs/plans/2026-07-22-initjoin-selfform-fallback.md
T
Joseph Doherty a61e041e58 feat(health)!: active node is the oldest Up member, not the leader (0.3.0) 
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one
shared primitive, and retires the leader/RoleLeader selection they were written
to avoid.

Both Akka consumers had already hand-rolled a replacement rather than use this
package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa
ClusterPrimaryHealthCheck), so the entire active-node surface here —
ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not
merely unused: it was avoided, twice, for the same reason.

Leadership is address-ordered (host, then port) and has no relationship to time;
singleton placement is age-ordered. The two agree on a freshly-formed cluster,
which is why single-node and happy-path tests never caught it. They diverge
permanently after any restart: the restarted node rejoins as the youngest but
keeps its address, so if it holds the lower address it becomes leader while the
singletons — and all the work they own — stay on the other node.

New ClusterActiveNode is the single implementation: oldest Up member, optional
role scope, plus role-preference resolution for a fused node that must answer
for the role its singletons are pinned to. ActiveNodeHealthCheck and
AkkaActiveNodeGate both delegate to it, so an endpoint gate and the
/health/active probe an orchestrator routes by cannot disagree.

BREAKING (behaviour, not signature):
- Selection is by age, not leadership.
- The role-filtered mode no longer reports Healthy for a node LACKING the role.
  That "not applicable => Healthy" made the tier answer 200 on every node and
  silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default,
  overridable via NoActiveRoleStatus. This case is reachable, not defensive —
  OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes.
- Identity compares UniqueAddress, so a node restarted on the same host:port is
  correctly a different member during the overlap.

Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data
object, so a standby reports WHO is active and a dashboard can render a pair
from either node's payload alone.

Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned
against a real two-node cluster built so the oldest member is not the
lowest-addressed one, with a fixture sanity check so it cannot pass for the
wrong reason.
2026-07-24 13:07:56 -04:00

12 KiB
Raw Blame History

InitJoin Self-Form Fallback (Cluster:SelfFormAfter) + Manual Failover Control — Shared Design & Index

For Claude: This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):

  • ScadaBridge: ~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
  • OtOpcUa: ~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md (7 tasks + "Reserved for additions" section)
  • OtOpcUa mesh-alignment PROGRAM (added 2026-07-22): ~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md.

Execute each with superpowers-extended-cc:executing-plans in that repo. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.

Goal: (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.

⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned

Both repo plans executed. The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores Join once joined" is FALSE — the watchdog sits outside Akka's join handshake, and a Join(self) fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a permanent second cluster at the same address. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.

  • ScadaBridge rejected the watchdog entirely and shipped self-first seed ordering (each node lists itself as seed-nodes[0]; Akka's FirstSeedNodeProcess provides the intended semantics race-free inside the handshake; StartupValidator hard-gates the ordering). No SelfFormAfter option exists there. See its plan's "Part 1 as executed".
  • OtOpcUa first kept SelfFormAfter behind a TCP reachability guard, then converged the same day (lmxopcua @ 3f24d4d6): ClusterBootstrapFallback + SelfFormAfter deleted, self-first ordering shipped with a conditional AkkaClusterOptionsValidator (self must be seed[0] only IF self is a seed — site nodes exempt; identity on PublicHostname+port, not the 0.0.0.0 bind).
  • Both repos now share the identical mechanism. Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
  • The design sections below are retained as the decision record — read them through this banner. akka_failover.md §6.1 carries the corrected family-level account.

Decision provenance: scadaproj/akka_failover.md §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory ha-availability-over-partition-safety). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.

Design

The defect being fixed

Akka.NET only allows the first-listed seed to self-join and form a new cluster. Every other node loops on InitJoin forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's ClusterOptions.SeedNodes doc comment even claims "either can start first and form the cluster" — which the deployed configs do not deliver (both nodes list node-a first). The plans make the claim true and fix the comment.

Behavior specification

Scenario Behavior with fallback
Peer alive (any boot order) Normal seed join within ms — fallback never fires (membership arrives before the window)
Lone cold-start, node IS in its own seed list After SelfFormAfter (default 10 s): warn log + Cluster.Join(SelfAddress) → Up alone, singletons start (min-nr-of-members=1)
Lone cold-start, node NOT in its own seed list Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. This is the OtOpcUa site-node topology (seeded only by central-1).
Peer boots after survivor self-formed Peer's InitJoin reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island.
Both cold-start simultaneously, mutually unreachable (boot-time partition) Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start.
SelfFormAfter = null or <= 0 Disabled — today's wait-forever behavior
Window expires while a join is mid-handshake Benign race: Akka ignores Join once the node has already joined a cluster

Config surface (same key both apps — family convention, like SplitBrainResolverStrategy)

  • ScadaBridge: ScadaBridge:Cluster:SelfFormAfter → ClusterOptions.SelfFormAfter (TimeSpan?, default 00:00:10), validated non-negative-or-null by ClusterOptionsValidator.
  • OtOpcUa: Cluster:SelfFormAfter → AkkaClusterOptions.SelfFormAfter (TimeSpan?, default 00:00:10); disable-on-<=0 semantics live in the fallback itself.
  • NOT a code constant (deliberate — see the DowningStableAfter lesson, akka_failover.md §6.2).

Mechanism (both apps, deliberately duplicated like the termination watchdogs)

ClusterBootstrapFallback.Arm(system, options, logger), called at ActorSystem creation (ScadaBridge: inside AkkaHostedService.GetOrCreateActorSystem; OtOpcUa: an Akka.Hosting AddStartup task inside WithOtOpcUaClusterBootstrap, so production and test hosts arm identically):

  1. SelfFormAfter null/≤0 → log info, return (disabled).
  2. Self address not in own SeedNodes (parsed via Address.Parse, exact equality) → log info, return (island guard).
  3. RegisterOnMemberUp completes a TCS; race it against Task.Delay(window).
  4. On window expiry (and system not terminated): prominent warn log + Cluster.Join(cluster.SelfAddress).

Races are benign — Akka ignores Join once joined this incarnation.

Manual failover control (Part D design)

Mechanism — graceful Cluster.Leave, never Down. "Trigger failover" = issue Cluster.Leave(<current active/primary member's address>) from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its CoordinatedShutdown runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the youngest member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.

Shared rules:

  • Admin-only: button renders inside the app's admin-policy AuthorizeView (ScadaBridge AuthorizationPolicies.RequireAdmin on /monitoring/health; OtOpcUa the mutating-pages policy from AdminUiPolicies on /clusters/{id}/redundancy).
  • Peer guard: disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
  • Confirmation dialog stating exactly what happens.
  • Audited through each app's existing audit seam before the Leave (cluster.manual-failover; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
  • UX note (ScadaBridge): Traefik routes the UI to the active node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
  • Scope note (OtOpcUa): until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in docs/Redundancy.md).
  • Election parity: each service's target query is intentionally identical to the app's active/primary election (ActiveNodeEvaluator.SelfIsOldestUp / RedundancyStateActor.SelectDriverPrimary), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
  • Self-form interplay: none — the restarted node rejoins via its peer (alive by definition here), so SelfFormAfter never fires on this path.

Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)

Verdict: usable in principle, deliberately NOT adopted. The MultiNode TestKit (Akka.MultiNode.TestAdapter + Akka.Remote.TestKit + Akka.Cluster.TestKit) runs each RoleName as a separate process under dotnet test, with TestConductor able to Blackhole links and Exit nodes, synchronized by EnterBarrier. Reasons not to use it here:

  1. Everything this feature needs is testable in-process with real ActorSystems built from production bootstrap code (ScadaBridge TwoNodeClusterFixture; OtOpcUa SplitBrainResolverActivationTests pattern). Same fidelity, no new runner.
  2. Heavy project-level cost: dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
  3. OtOpcUa xunit constraint: the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
  4. The one MNTR-only scenario (true boot-time partition → dual self-form) validates an already-documented accepted risk; the family's preferred proof is the docker rig drill (failover-drill.sh precedent, or docker network disconnect).

Follow-up: revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.

Out of scope (recorded follow-ups)

  • Making OtOpcUa's 15 s DowningStableAfter configurable — now listed in the OtOpcUa plan's "Reserved for additions".
  • Failure-detector tuning knobs for faster failover (both apps) — see akka_failover.md §2 tuning table; OtOpcUa side listed in its reserved-additions section.
  • Staggered Windows service start delays (pure ops; document only).
  • deploy/wonder-app-vd03/ gitignored production overlay (ops applies SelfFormAfter on next deploy).
  • ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).

Task C1: Family docs flip (execute AFTER both repo plans complete and merge)

Classification: trivial Estimated implement time: ~2 min

Files:

  • Modify: ~/Desktop/scadaproj/akka_failover.md §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 DowningStableAfter line unless the OtOpcUa reserved-addition shipped.
  • Modify: ~/Desktop/scadaproj/CLAUDE.md — brief addition to the ScadaBridge and OtOpcUa index rows.
  • Update memory ha-availability-over-partition-safety.md — mark the decision implemented.
Reference in New Issue View Git Blame Copy Permalink