diff --git a/docs/plans/2026-07-21-per-cluster-mesh-design.md b/docs/plans/2026-07-21-per-cluster-mesh-design.md index addeb661..4b83db54 100644 --- a/docs/plans/2026-07-21-per-cluster-mesh-design.md +++ b/docs/plans/2026-07-21-per-cluster-mesh-design.md @@ -2,8 +2,8 @@ > **Status:** design for review. No implementation planned yet. > **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it, -> the two-node keep-oldest gap accepted as ScadaBridge accepts it, and their auth posture matched for -> now. +> the two-node keep-oldest gap CLOSED upstream and still live here (§6.2 — most urgent item), and +> their auth posture matched for now. ## 1. What this changes @@ -209,27 +209,33 @@ SHA-256 verification, newest-2 retention and pair replication all carry over unc under this model it stops existing for driver nodes. That is a behaviour change on a client-visible value, not just an internal refactor. -### 6.2 DECIDED — accept the two-node keep-oldest gap, as ScadaBridge does +### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it -**Status corrected.** What was resolved is the *documentation and the drill*, not the gap. Gitea -**PR #12** (closed) — — "R2-01: Cluster, -Host & Failover round-2 fixes" carried: +**An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as +still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR +[#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill +*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap +in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner +decision 2026-07-21: availability over partition-safety)"*. -- **T1** (`badc97af`) — the failover drill now kills the STANDBY by default, and its `active` mode - *"measures the registered keep-oldest outage instead of pretending recovery"*. -- **T2** (`d5364506`) — per-direction failover docs, dropping what the commit message calls *"the - undeliverable ~25s active-crash promise"*. +Its commit message carries the analysis, live-proven on their rig: -So the closed work made the gap **honest**, not absent. `Component-ClusterInfrastructure.md:113` -still reads: *"Removing the gap itself is the **registered deferred keep-oldest topology/strategy -decision** (master tracker 2026-07-08, owner: user)."* There is no open Gitea issue for it — it is -tracked on that master tracker, not in the issue list. +> *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62 +> `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1 +> survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision … +> including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`); +> `keep-majority` just re-keys the fatal crash to the lowest address."* -OtOpcUa inherits the same gap and, for now, the same posture. Worth recording so it is a known -accepted risk rather than a surprise: after the oldest node of a two-node mesh dies, the survivor -self-downs and cannot re-bootstrap alone, because only the first-listed seed may self-join. Recovery -is operator-driven — restart the dead first seed, or restart the survivor with a self-first seed -override. +**OtOpcUa is running that exact configuration today** — +`Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`, +`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with +`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`. + +So a two-node OtOpcUa pair **cannot fail over when the oldest node crashes**: the survivor downs +itself. That is a total-outage defect in the redundancy feature itself, it is independent of this +design, and the sister project has already done the analysis and shipped the fix. **It should be +Phase 0 alongside the oldest-Up role derivation**, and it needs its own live gate — a +crash-the-oldest drill, since the failure only appears in the 1-vs-1 case. ### 6.3 DECIDED — match ScadaBridge's auth posture for now @@ -244,7 +250,8 @@ Deliberately not a task plan — per-phase plans follow, one at a time. | Phase | Content | Independent of the split? | |---|---|---| -| 0 | Oldest-Up role derivation (§4) | **Yes** — ship first | +| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — most urgent | +| 0b | Oldest-Up role derivation (§4) | **Yes** | | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |