Joseph Doherty
a61e041e58
Promotes the two apps' private replacements for ActiveNodeHealthCheck into one shared primitive, and retires the leader/RoleLeader selection they were written to avoid. Both Akka consumers had already hand-rolled a replacement rather than use this package (ScadaBridge OldestNodeActiveHealthCheck, OtOpcUa ClusterPrimaryHealthCheck), so the entire active-node surface here — ActiveNodeHealthCheck, AkkaActiveNodeGate — had ZERO consumers. It was not merely unused: it was avoided, twice, for the same reason. Leadership is address-ordered (host, then port) and has no relationship to time; singleton placement is age-ordered. The two agree on a freshly-formed cluster, which is why single-node and happy-path tests never caught it. They diverge permanently after any restart: the restarted node rejoins as the youngest but keeps its address, so if it holds the lower address it becomes leader while the singletons — and all the work they own — stay on the other node. New ClusterActiveNode is the single implementation: oldest Up member, optional role scope, plus role-preference resolution for a fused node that must answer for the role its singletons are pinned to. ActiveNodeHealthCheck and AkkaActiveNodeGate both delegate to it, so an endpoint gate and the /health/active probe an orchestrator routes by cannot disagree. BREAKING (behaviour, not signature): - Selection is by age, not leadership. - The role-filtered mode no longer reports Healthy for a node LACKING the role. That "not applicable => Healthy" made the tier answer 200 on every node and silently broke leader-pinning (lmxopcua#494); it is now Unhealthy by default, overridable via NoActiveRoleStatus. This case is reachable, not defensive — OtOpcUa's RoleParser admits dev-only and cluster-role-only nodes. - Identity compares UniqueAddress, so a node restarted on the same host:port is correctly a different member during the overlap. Results now carry activeRole/selfAddress/activeNode in the 0.2.0 per-entry data object, so a standby reports WHO is active and a dashboard can render a pair from either node's payload alone. Tests: 45 Akka (was 39), 76 total. The age-vs-address divergence is pinned against a real two-node cluster built so the oldest member is not the lowest-addressed one, with a fixture sanity check so it cannot pass for the wrong reason.
105 lines
12 KiB
Markdown
105 lines
12 KiB
Markdown
# InitJoin Self-Form Fallback (`Cluster:SelfFormAfter`) + Manual Failover Control — Shared Design & Index
|
||
|
||
> **For Claude:** This is the family-level DESIGN + INDEX document. The executable per-repo plans live in the repos (split 2026-07-22 — the OtOpcUa plan is expected to grow and has a reserved-additions section):
|
||
>
|
||
> - **ScadaBridge:** `~/Desktop/ScadaBridge/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (10 tasks: fallback 1–7, manual-failover UI 8–9, optional site-pair failover 10)
|
||
> - **OtOpcUa:** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md` (7 tasks + "Reserved for additions" section)
|
||
> - **OtOpcUa mesh-alignment PROGRAM (added 2026-07-22):** `~/Desktop/OtOpcUa/docs/plans/2026-07-22-per-cluster-mesh-program.md` — Phases 1–7 moving OtOpcUa to ScadaBridge's mesh shape (one 2-node Akka mesh per application Cluster, ClusterClient/gRPC/fetch transports, driver nodes off the ConfigDb), driven by the co-location decision: OtOpcUa pairs run on the SAME Windows VMs as the ScadaBridge nodes. The selfform plan above is its prerequisite. Design: `OtOpcUa/docs/plans/2026-07-21-per-cluster-mesh-design.md`.
|
||
>
|
||
> Execute each with superpowers-extended-cc:executing-plans **in that repo**. The two plans are fully independent and can run in parallel. Only Task C1 below executes from this file.
|
||
|
||
**Goal:** (1) Either node of a 2-node pair can cold-start alone and become operational, unattended — closing the seed-node bootstrap gap in both ScadaBridge and OtOpcUa via a configurable, fast (default 10 s) InitJoin timeout that falls back to self-forming a cluster. (2) An admin-only "Trigger failover" control on each app's cluster-status page for operator-initiated, graceful role swaps.
|
||
|
||
> ## ⚠️ EXECUTION OUTCOME (2026-07-22) — design partially overturned
|
||
>
|
||
> Both repo plans executed. **The behavior-spec row "Window expires while a join is mid-handshake → benign race: Akka ignores `Join` once joined" is FALSE** — the watchdog sits outside Akka's join handshake, and a `Join(self)` fired while a join is in flight (routine restart into a live peer, join stalled behind stale-incarnation removal) abandons the join and forms a **permanent second cluster at the same address**. Proven independently by a ScadaBridge test (split unhealed after 90 s) and by an OtOpcUa live island during the manual-failover drill.
|
||
>
|
||
> - **ScadaBridge** rejected the watchdog entirely and shipped **self-first seed ordering** (each node lists itself as `seed-nodes[0]`; Akka's `FirstSeedNodeProcess` provides the intended semantics race-free inside the handshake; `StartupValidator` hard-gates the ordering). No `SelfFormAfter` option exists there. See its plan's "Part 1 as executed".
|
||
> - **OtOpcUa** first kept `SelfFormAfter` behind a TCP reachability guard, then **converged the same day** (`lmxopcua @ 3f24d4d6`): `ClusterBootstrapFallback` + `SelfFormAfter` deleted, self-first ordering shipped with a conditional `AkkaClusterOptionsValidator` (self must be `seed[0]` only IF self is a seed — site nodes exempt; identity on `PublicHostname`+port, not the 0.0.0.0 bind).
|
||
> - **Both repos now share the identical mechanism.** Everything merged + pushed 2026-07-22, including ScadaBridge Task 10 (site-pair failover relayed from the central UI).
|
||
> - The design sections below are retained as the decision record — read them through this banner. `akka_failover.md` §6.1 carries the corrected family-level account.
|
||
|
||
**Decision provenance:** `scadaproj/akka_failover.md` §6.1 (decided 2026-07-22); design priority = availability over partition safety (memory `ha-availability-over-partition-safety`). Nodes share a datacenter/LAN → a live peer answers InitJoin in milliseconds, so 10 s is generous; operators tune per site.
|
||
|
||
---
|
||
|
||
## Design
|
||
|
||
### The defect being fixed
|
||
|
||
Akka.NET only allows the **first-listed seed** to self-join and form a *new* cluster. Every other node loops on `InitJoin` forever until some existing member answers. Both repos document this as the "registered outage gap": a non-first seed cold-starting while the first seed is down never becomes Up (recovery today = manual env-var override or reviving the first seed). ScadaBridge's `ClusterOptions.SeedNodes` doc comment even claims "either can start first and form the cluster" — which the deployed configs do **not** deliver (both nodes list node-a first). The plans make the claim true and fix the comment.
|
||
|
||
### Behavior specification
|
||
|
||
| Scenario | Behavior with fallback |
|
||
|---|---|
|
||
| Peer alive (any boot order) | Normal seed join within ms — fallback never fires (membership arrives before the window) |
|
||
| Lone cold-start, node IS in its own seed list | After `SelfFormAfter` (default 10 s): warn log + `Cluster.Join(SelfAddress)` → Up alone, singletons start (`min-nr-of-members=1`) |
|
||
| Lone cold-start, node NOT in its own seed list | Fallback inert (info log at arm time); node waits for a seed — self-forming would island it. **This is the OtOpcUa site-node topology** (seeded only by `central-1`). |
|
||
| Peer boots after survivor self-formed | Peer's `InitJoin` reaches the self-formed node (it's in the peer's seed list) → joins as youngest/standby. No island. |
|
||
| Both cold-start simultaneously, mutually unreachable (boot-time partition) | Both self-form → two islands (same dual-active class auto-down already accepts; same recovery: restart one side). Mitigable with a staggered service start. |
|
||
| `SelfFormAfter` = `null` or `<= 0` | Disabled — today's wait-forever behavior |
|
||
| Window expires while a join is mid-handshake | Benign race: Akka ignores `Join` once the node has already joined a cluster |
|
||
|
||
### Config surface (same key both apps — family convention, like `SplitBrainResolverStrategy`)
|
||
|
||
- **ScadaBridge:** `ScadaBridge:Cluster:SelfFormAfter` → `ClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`), validated non-negative-or-null by `ClusterOptionsValidator`.
|
||
- **OtOpcUa:** `Cluster:SelfFormAfter` → `AkkaClusterOptions.SelfFormAfter` (`TimeSpan?`, default `00:00:10`); disable-on-`<=0` semantics live in the fallback itself.
|
||
- NOT a code constant (deliberate — see the `DowningStableAfter` lesson, `akka_failover.md` §6.2).
|
||
|
||
### Mechanism (both apps, deliberately duplicated like the termination watchdogs)
|
||
|
||
`ClusterBootstrapFallback.Arm(system, options, logger)`, called at ActorSystem creation (ScadaBridge: inside `AkkaHostedService.GetOrCreateActorSystem`; OtOpcUa: an Akka.Hosting `AddStartup` task inside `WithOtOpcUaClusterBootstrap`, so production and test hosts arm identically):
|
||
|
||
1. `SelfFormAfter` null/`≤0` → log info, return (disabled).
|
||
2. Self address not in own `SeedNodes` (parsed via `Address.Parse`, exact equality) → log info, return (island guard).
|
||
3. `RegisterOnMemberUp` completes a TCS; race it against `Task.Delay(window)`.
|
||
4. On window expiry (and system not terminated): prominent warn log + `Cluster.Join(cluster.SelfAddress)`.
|
||
|
||
Races are benign — Akka ignores `Join` once joined this incarnation.
|
||
|
||
### Manual failover control (Part D design)
|
||
|
||
**Mechanism — graceful `Cluster.Leave`, never `Down`.** "Trigger failover" = issue `Cluster.Leave(<current active/primary member's address>)` from whichever node serves the UI (any member may initiate a leave for any member). The leaving node walks the graceful path: singleton hand-over via the cluster-leave phases (ScadaBridge additionally drains each singleton 10 s), its `CoordinatedShutdown` runs, the existing termination watchdog exits the process, the service supervisor restarts it, and it rejoins as the **youngest** member — a permanent role swap under the oldest-Up election, with no downing window and no dead interval beyond hand-over latency.
|
||
|
||
Shared rules:
|
||
- **Admin-only:** button renders inside the app's admin-policy `AuthorizeView` (ScadaBridge `AuthorizationPolicies.RequireAdmin` on `/monitoring/health`; OtOpcUa the mutating-pages policy from `AdminUiPolicies` on `/clusters/{id}/redundancy`).
|
||
- **Peer guard:** disabled (with tooltip) when fewer than 2 Up members carry the relevant role.
|
||
- **Confirmation dialog** stating exactly what happens.
|
||
- **Audited** through each app's existing audit seam before the Leave (`cluster.manual-failover`; ScadaBridge uses its CENTRAL audit writer, not the shared seam — dual-seam gotcha).
|
||
- **UX note (ScadaBridge):** Traefik routes the UI to the *active* node, so triggering central failover drops the Blazor circuit; the page reconnects against the new active. The dialog warns about this.
|
||
- **Scope note (OtOpcUa):** until the per-cluster mesh lands, the driver-Primary election is mesh-wide — the button acts on THE Primary of the whole Akka cluster; the panel says so (mirrors the KNOWN LIMITATION in `docs/Redundancy.md`).
|
||
- **Election parity:** each service's target query is intentionally identical to the app's active/primary election (`ActiveNodeEvaluator.SelfIsOldestUp` / `RedundancyStateActor.SelectDriverPrimary`), pinned by a parity test — the node acted on must be exactly the one hosting the singletons.
|
||
- **Self-form interplay:** none — the restarted node rejoins via its peer (alive by definition here), so `SelfFormAfter` never fires on this path.
|
||
|
||
### Multi-node testing assessment (getakka.net/articles/testing/multi-node-testing.html)
|
||
|
||
**Verdict: usable in principle, deliberately NOT adopted.** The MultiNode TestKit (`Akka.MultiNode.TestAdapter` + `Akka.Remote.TestKit` + `Akka.Cluster.TestKit`) runs each `RoleName` as a separate process under `dotnet test`, with `TestConductor` able to `Blackhole` links and `Exit` nodes, synchronized by `EnterBarrier`. Reasons not to use it here:
|
||
|
||
1. **Everything this feature needs is testable in-process** with real ActorSystems built from production bootstrap code (ScadaBridge `TwoNodeClusterFixture`; OtOpcUa `SplitBrainResolverActivationTests` pattern). Same fidelity, no new runner.
|
||
2. **Heavy project-level cost:** dedicated test project, parallelization disabled assembly-wide, single-public-constructor constraints, MNTR log plumbing.
|
||
3. **OtOpcUa xunit constraint:** the Akka TestKit family is xunit-v2-only; OtOpcUa's integration tree is xunit.v3.
|
||
4. **The one MNTR-only scenario** (true boot-time partition → dual self-form) validates an already-documented *accepted* risk; the family's preferred proof is the docker rig drill (`failover-drill.sh` precedent, or `docker network disconnect`).
|
||
|
||
**Follow-up:** revisit alongside OtOpcUa per-cluster-mesh Phase 7 (failover-drill phase) if a scripted dual-island drill is ever wanted.
|
||
|
||
### Out of scope (recorded follow-ups)
|
||
|
||
- Making OtOpcUa's 15 s `DowningStableAfter` configurable — now listed in the OtOpcUa plan's "Reserved for additions".
|
||
- Failure-detector tuning knobs for faster failover (both apps) — see `akka_failover.md` §2 tuning table; OtOpcUa side listed in its reserved-additions section.
|
||
- Staggered Windows service start delays (pure ops; document only).
|
||
- `deploy/wonder-app-vd03/` gitignored production overlay (ops applies `SelfFormAfter` on next deploy).
|
||
- ScadaBridge site-pair failover from the central UI — optional Task 10 in the ScadaBridge plan (new central→site message contract; confirm with the user first).
|
||
|
||
---
|
||
|
||
## Task C1: Family docs flip (execute AFTER both repo plans complete and merge)
|
||
|
||
**Classification:** trivial
|
||
**Estimated implement time:** ~2 min
|
||
|
||
**Files:**
|
||
- Modify: `~/Desktop/scadaproj/akka_failover.md` §6.1 — "Status: decided, not yet implemented" → implemented (merge commit refs from both repos); leave the §6.2 `DowningStableAfter` line unless the OtOpcUa reserved-addition shipped.
|
||
- Modify: `~/Desktop/scadaproj/CLAUDE.md` — brief addition to the ScadaBridge and OtOpcUa index rows.
|
||
- Update memory `ha-availability-over-partition-safety.md` — mark the decision implemented.
|