dohertj2/lmxopcua
1
0
Fork 0
Code Issues 9 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs(mesh): record phases 0a/0b as done and plan phase 1

Browse Source 
The design doc still described the downing gap and the RoleLeader derivation as
open. Both shipped earlier today, so 6.2 and 4 now carry the fix and its pin
instead of the diagnosis, and 7 marks 0a/0b done. 0a's live gate stays open and
is explicitly deferred to phase 7: the failure only appears in a 1-vs-1 split and
docker-dev is a single six-node mesh, so there is no rig on which to drill it
until the meshes are actually split.

The phase 1 plan lands with one finding that changes its shape. The design pairs
"ClusterNode gains address columns" with "coordinator sources its expected-ack set
from the DB" as though they were one change; they are independent, and the second
is far smaller than it reads. ClusterNode.NodeId is ALREADY in the coordinator's
identifier space — the rig seeds central-1:4053 and so on, because
NodeDeploymentState.NodeId is FK-bound to it and the coordinator's
membership-derived ids already satisfy that FK. That the deploy path works today
is standing proof the two sets agree.

What is not small is the semantics. Membership-derived and DB-derived sets differ
on a configured-but-stopped node: today the deployment seals green without it —
telling the operator the fleet is deployed when it is not — and afterwards it
would fail at the apply deadline. Failing is the better behaviour and is required
once central cannot see cluster membership, but it needs an escape hatch or a node
down for maintenance blocks every deploy. ClusterNode.Enabled already is that
hatch, so the plan filters on it and gates the phase on a live check that proves
both halves: a stopped node fails the deploy legibly, and disabling it lets the
deploy seal.

The plan also adds a task the design does not have: AkkaPort duplicates the node's
own Cluster:Port with nothing making them agree, and an unenforced duplicated
address is the same shape as the Modbus/ModbusTcp and TwinCat/Focas drifts already
recorded here.

Phases 2-7 remain unstarted. They change the running data path and rewrite the
rig, and each needs its own plan and live gate.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:46:10 -04:00
parent b2d19a1730
commit 84c6524edc
2 changed files with 217 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-07-21-per-cluster-mesh-design.md
+26 -10
View File
@@ -1,9 +1,10 @@
# Per-Cluster Akka Mesh — design
> **Status:** design for review. No implementation planned yet.
> **Status:** design for review. **Phases 0a and 0b are DONE and merged** (2026-07-21); Phases 1–7
> are not started and each needs its own plan.
> **Decisions settled 2026-07-21** — see §6: one central DB with driver nodes disconnected from it,
> the two-node keep-oldest gap CLOSED upstream and still live here (§6.2 — most urgent item), and
> their auth posture matched for now.
> the two-node keep-oldest gap closed upstream and now closed here too (§6.2), and their auth posture
> matched for now.
## 1. What this changes
@@ -142,6 +143,12 @@ the dual-primary shape archreview 03/S4 exists to prevent.
and should not wait for this design.** It also happens to be exactly what the separate-mesh model
needs, so it is not throwaway work.
**FIXED 2026-07-21 (Phase 0b).** `RedundancyStateActor.BuildSnapshot` selects the oldest Up member
carrying the `driver` role, matching singleton placement; `NodeRedundancyState.IsRoleLeaderForDriver`
was renamed `IsDriverPrimary` so the contract stops asserting the old derivation. Pinned by
`RedundancyPrimaryElectionTests`, which builds a real two-node cluster in which the oldest node holds
the *higher* address, so the two orderings genuinely disagree.
## 5. Target architecture
Mirror ScadaBridge's shape, adapted to OtOpcUa's existing substrate:
@@ -231,11 +238,20 @@ Its commit message carries the analysis, live-proven on their rig:
`keep-oldest { down-if-alone = on }`, `stable-after = 15s`, with
`BuildClusterOptions` setting the matching typed `KeepOldestOption { DownIfAlone = true }`.
So a two-node OtOpcUa pair **cannot fail over when the oldest node crashes**: the survivor downs
itself. That is a total-outage defect in the redundancy feature itself, it is independent of this
design, and the sister project has already done the analysis and shipped the fix. **It should be
Phase 0 alongside the oldest-Up role derivation**, and it needs its own live gate — a
crash-the-oldest drill, since the failure only appears in the 1-vs-1 case.
So a two-node OtOpcUa pair **could not fail over when the oldest node crashed**: the survivor
downed itself. That was a total-outage defect in the redundancy feature itself, independent of this
design.
**FIXED 2026-07-21 (Phase 0a).** `Cluster:SplitBrainResolverStrategy` now selects the downing
provider and defaults to `auto-down` — Akka's `AutoDowning` with a 15s window — so the leader among
the reachable members downs the unreachable peer and a crash of either node fails over. `keep-oldest`
remains selectable for deployments that prefer partition-safety. Pinned by
`SplitBrainResolverActivationTests`, which asserts the provider read back off a running ActorSystem
rather than the configured one. See `docs/Redundancy.md`.
**Its live gate is still outstanding**, and cannot run on today's rig: the failure only appears in a
1-vs-1 split, and `docker-dev` is a single six-node mesh. The crash-the-oldest drill therefore lands
with Phase 6/7, which makes every mesh exactly two nodes.
### 6.3 DECIDED — match ScadaBridge's auth posture for now
@@ -250,8 +266,8 @@ Deliberately not a task plan — per-phase plans follow, one at a time.
| Phase | Content | Independent of the split? |
|---|---|---|
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — most urgent |
| 0b | Oldest-Up role derivation (§4) | **Yes** |
| 0a | Downing strategy: keep-oldest cannot survive an oldest-node crash in a 2-node cluster (§6.2) | **Yes** — **DONE 2026-07-21**, live gate deferred to Phase 7 |
| 0b | Oldest-Up role derivation (§4) | **Yes** — **DONE 2026-07-21** |
| 1 | `ClusterNode` gains Akka + gRPC address columns; coordinator sources its expected-ack set from the DB | Yes |
| 2 | Comm actors + receptionist registration; ClusterClient transport; deploy notify + acks across the boundary | No |
| 3 | Config fetch-and-cache: artifact served by central, driver nodes read LocalDb (§6.1) | No |