docs(design): correct 6.2 — the keep-oldest gap is closed upstream and live here
v2-ci / build (push) Failing after 1m35s
v2-ci / unit-tests (push) Has been skipped

I reported this gap as still open, citing a requirements doc line and the PR that
made the failover drill honest about it. Both were stale: ScadaBridge closed it
three hours earlier in cf3bd52f by switching to auto-down. My check missed it
because I compared only one direction (main..origin/main) and never asked whether
local main was AHEAD of origin, so two unpushed commits were invisible to me.

The substance matters more than the correction. Their commit message records a
live-proven finding: Akka.NET 1.5.62 KeepOldest.OldestDecision only lets
down-if-alone rescue a side with >= 2 members, so in a 1-vs-1 split the survivor
takes DownReachable and downs ITSELF. OtOpcUa runs precisely that configuration —
akka.conf active-strategy keep-oldest with down-if-alone on, plus the matching
typed KeepOldestOption — which means a two-node pair cannot fail over when the
oldest node crashes. That is a total-outage defect inside the redundancy feature,
independent of the mesh design, and it is now Phase 0a: most urgent, and needing a
crash-the-oldest live gate because the failure only appears 1-vs-1.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 14:16:18 -04:00
parent 8838e92d0a
commit c6fb8bb0ad
@@ -2,8 +2,8 @@
> **Status:** design for review. No implementation planned yet. > **Status:** design for review. No implementation planned yet.
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it, > **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap accepted as ScadaBridge accepts it, and their auth posture matched for > the two-node keep-oldest gap CLOSED upstream and still live here (§6.2 — most urgent item), and
> now. > their auth posture matched for now.
## 1. What this changes ## 1. What this changes
@@ -209,27 +209,33 @@ SHA-256 verification, newest-2 retention and pair replication all carry over unc
under this model it stops existing for driver nodes. That is a behaviour change on a client-visible under this model it stops existing for driver nodes. That is a behaviour change on a client-visible
value, not just an internal refactor. value, not just an internal refactor.
### 6.2 DECIDED — accept the two-node keep-oldest gap, as ScadaBridge does ### 6.2 CORRECTED — the gap was closed in ScadaBridge, and OtOpcUa still has it
**Status corrected.** What was resolved is the *documentation and the drill*, not the gap. Gitea **An earlier revision of this section was wrong.** It reported the two-node keep-oldest outage gap as
**PR #12** (closed) — <https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12> — "R2-01: Cluster, still open, citing `Component-ClusterInfrastructure.md:113` and Gitea PR
Host & Failover round-2 fixes" carried: [#12](https://gitea.dohertylan.com/dohertj2/ScadaBridge/pulls/12) (which made the failover drill
*honest* about the gap rather than closing it). That doc line was stale: ScadaBridge closed the gap
in `cf3bd52f`, *"feat(cluster): auto-down downing strategy — either-node crash now fails over (owner
decision 2026-07-21: availability over partition-safety)"*.
- **T1** (`badc97af`) — the failover drill now kills the STANDBY by default, and its `active` mode Its commit message carries the analysis, live-proven on their rig:
*"measures the registered keep-oldest outage instead of pretending recovery"*.
- **T2** (`d5364506`) — per-direction failover docs, dropping what the commit message calls *"the
undeliverable ~25s active-crash promise"*.
So the closed work made the gap **honest**, not absent. `Component-ClusterInfrastructure.md:113` > *"Two-node keep-oldest could NEVER survive a crash of the oldest/active node: Akka.NET 1.5.62
still reads: *"Removing the gap itself is the **registered deferred keep-oldest topology/strategy > `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side with >= 2 members, so the 1-vs-1
decision** (master tracker 2026-07-08, owner: user)."* There is no open Gitea issue for it — it is > survivor takes `DownReachable` and downs ITSELF — proven live on the rig ('SBR took decision …
tracked on that master tracker, not in the issue list. > including myself') before this change. `static-quorum(1)` is worse (`IsTooManyMembers` -> `DownAll`);
> `keep-majority` just re-keys the fatal crash to the lowest address."*
OtOpcUa inherits the same gap and, for now, the same posture. Worth recording so it is a known **OtOpcUa is running that exact configuration today** —
accepted risk rather than a surprise: after the oldest node of a two-node mesh dies, the survivor `Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf:49-53`: `active-strategy = "keep-oldest"`,
self-downs and cannot re-bootstrap alone, because only the first-listed seed may self-join. Recovery `keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
is operator-driven — restart the dead first seed, or restart the survivor with a self-first seed `BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
override.
So a two-node OtOpcUa pair **cannot fail over when the oldest node crashes**: the survivor downs
itself. That is a total-outage defect in the redundancy feature itself, it is independent of this
design, and the sister project has already done the analysis and shipped the fix. **It should be
Phase 0 alongside the oldest-Up role derivation**, and it needs its own live gate — a
crash-the-oldest drill, since the failure only appears in the 1-vs-1 case.
### 6.3 DECIDED — match ScadaBridge's auth posture for now ### 6.3 DECIDED — match ScadaBridge's auth posture for now
@@ -244,7 +250,8 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
| Phase | Content | Independent of the split? | | Phase | Content | Independent of the split? |
|---|---|---| |---|---|---|
| 0 | Oldest-Up role derivation (§4) | **Yes** — ship first | | 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — most urgent |
| 0b | Oldest-Up role derivation (§4) | **Yes** |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes | | 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No | | 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No | | 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |