dohertj2/lmxopcua
1
0
Fork 0
Code Issues 9 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(redundancy): SelfFormAfter closes the seed-node bootstrap gap

Browse Source 
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 06:10:11 -04:00
parent b8ac7e35e6
commit b853185cfd
7 changed files with 666 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+2
View File
@@ -224,6 +224,8 @@ The server supports non-transparent warm/hot redundancy via the `Redundancy` sec
- **Downing is `auto-down`, not `keep-oldest`.** `Cluster:SplitBrainResolverStrategy` (default `auto-down`) selects the provider via `ServiceCollectionExtensions.BuildDowningHocon`; an unknown value fails host start. A two-node pair running the previous `keep-oldest` + `down-if-alone` **could not survive a crash of the oldest node**: Akka's `KeepOldest.OldestDecision` only lets `down-if-alone` rescue a side holding ≥ 2 members, so the 1-vs-1 survivor downs *itself* and exits. `down-if-alone` is a 3+-node feature. The accepted trade for `auto-down` is dual-active during a genuine network partition. **Its live gate is still open** — the pathology only appears 1-vs-1 and docker-dev is a single six-node mesh, so the crash-the-oldest drill is deferred to the per-cluster mesh work.
- **The Primary is the oldest Up `driver` member, not the role leader.** `RedundancyStateActor` previously used `ClusterState.RoleLeader("driver")` (lowest *address*), while `ClusterSingletonManager` places singletons on the *oldest*. They agree only on a freshly-formed cluster and diverge after any restart, so the Primary-gated data plane (writes, alarm acks, alerts emit, alarm-history drain) could enable on a node not hosting the singletons. `NodeRedundancyState.IsRoleLeaderForDriver` was renamed **`IsDriverPrimary`**.
**A third bootstrap gap closed 2026-07-22 — downing was never the whole story.** Even under `auto-down`, a node cold-starting while its peer was dead **never came Up**: Akka lets only the **first** listed seed form a *new* cluster, so every other node loops on `InitJoin` forever. `Cluster:SelfFormAfter` (`TimeSpan?`, default 10 s, `null`/`≤0` disables) arms `ClusterBootstrapFallback` from `WithOtOpcUaClusterBootstrap` via an Akka.Hosting `AddStartup` task — it waits that long for membership, then `Cluster.Join(SelfAddress)`. **It fires only when this node's own address is in its own `SeedNodes`**: a node seeded only by someone else (today's docker-dev site nodes, which list only `central-1`) must wait rather than island itself permanently. Pinned by `SelfFormBootstrapTests`, whose two negative cases each carry a positive-control self-join so they cannot pass vacuously. See `docs/Redundancy.md` §"Bootstrap: the self-form fallback".
Assert the **effective** cluster config off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never the typed options or the akka.conf text — several HOCON fragments compete and the losing one still reads correctly in the file.
**Still per-Akka-cluster, not per application `Cluster`.** One Primary is elected across the whole Akka mesh, so a fleet running several application clusters in one mesh (what docker-dev models) gets one Primary among all driver nodes. The fix is the per-cluster mesh workstream: `docs/plans/2026-07-21-per-cluster-mesh-design.md` (Phases 0a/0b done; 1–7 not started, Phase 1 planned in `…-phase1.md`).