dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(cluster): self-first seed ordering replaces the self-form fallback
v2-ci / build (push) Successful in 3m44s
Details
v2-ci / unit-tests (push) Failing after 10m7s
Details

Browse Source 
docs/Redundancy.md's bootstrap section is rewritten around the mechanism that is
actually in force: which of Akka's two bootstrap processes runs is decided by
whether seed-nodes[0] is this node's own address, so the ORDER of Cluster:SeedNodes
is the fix, not a timer. Adds the per-process table, the shipped self-first
example, why the validator is conditional (site nodes are not seeds) and why it
matches PublicHostname rather than the 0.0.0.0 bind address.

The retirement is documented rather than erased: a new subsection explains that a
watchdog outside the join handshake cannot tell "no seed answered" from "a seed
answered and the join is in flight", and the ea45ace1 live-gate finding is kept
verbatim as the evidence that produced that conclusion (InitJoinAck at 10:49:05,
old incarnation retired 10:49:06, watchdog fired 10:49:15, second cluster). The
watchdog's own earlier PASS is kept too — it did pass what it was gated on.

Also: a live gate for the NEW mechanism is registered as outstanding (not
re-drilled on the docker-dev rig since the swap), and the plan doc that shipped
the watchdog gets a superseded banner rather than an edit, so the execution record
and its Task 8 finding stay readable.

Ripple: CLAUDE.md cluster section, docs/Configuration.md + docs/v2/Cluster.md +
docs/ServiceHosting.md SeedNodes entries, the per-cluster-mesh design doc's two
"CLOSED by SelfFormAfter" notes, and mesh-program Phase 6 (which had this
convergence queued — now marked done early) + Phase 7's live-gate name.
This commit is contained in:
Joseph Doherty
2026-07-22 07:46:02 -04:00
parent 3f24d4d6bf
commit 9af935f237
9 changed files with 132 additions and 81 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Redundancy.md
+81 -57
View File
@@ -143,13 +143,18 @@ The admin singleton is the cluster's only `RedundancyStateActor`. If the admin l
Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
```json
```jsonc
{
"Cluster": {
"Hostname": "0.0.0.0",
"Port": 4053,
"PublicHostname": "node-a.lan",
"SeedNodes": ["akka.tcp://otopcua@node-a.lan:4053"],
// Self FIRST, partner second — see "Bootstrap: self-first seed ordering".
// node-b.lan's own config lists node-b.lan first and node-a.lan second.
"SeedNodes": [
"akka.tcp://otopcua@node-a.lan:4053",
"akka.tcp://otopcua@node-b.lan:4053"
],
"Roles": ["admin", "driver"]
}
}
@@ -159,7 +164,7 @@ Per-node identity comes from `appsettings.json` + the `OTOPCUA_ROLES` env var:
OTOPCUA_ROLES=admin,driver
```
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. The first node bootstraps the cluster (its address goes in `SeedNodes`); the second node joins via the same `SeedNodes` list. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — see [Bootstrap: the self-form fallback](#bootstrap-the-self-form-fallback) for why the seed list alone is not enough, and what `Cluster:SelfFormAfter` adds.
Both nodes share the same `ConfigDb` connection string; `Cluster.PublicHostname` + `Roles` are what makes them distinct in cluster gossip. List **both** peers in `SeedNodes` on **both** nodes so either can cold-start alone — **and list each node ITSELF first**: the order decides which Akka bootstrap process runs, and a node that lists its partner first can never form the cluster while that partner is down. See [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
There is no longer a `Node:NodeId` setting and no `ClusterNode.RedundancyRole` column (the V2 migration dropped it — primary/secondary is now derived from cluster membership age). NodeId is derived as `host:port` of the cluster `PublicHostname` (see `ClusterRoleInfo.LocalNode` for the formula).
@@ -241,9 +246,10 @@ two-node pair that choice means *any* crash of the oldest node is a full outage.
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
(it is never the one downed); the **downed** node still does.
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer —
plus the **self-form fallback** below, without which a node cold-starting while its peer is dead waits
in `InitJoin` forever (auto-down removes the crash outage, not that one).
3. **Both peers listed in `SeedNodes`** on every node, **each node listing itself first**, so a restarted
node can re-join via either peer and a node cold-starting while its peer is dead still forms the
cluster. Without the self-first ordering it waits in `InitJoin` forever (auto-down removes the crash
outage, not that one) — see [Bootstrap: self-first seed ordering](#bootstrap-self-first-seed-ordering).
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
@@ -297,66 +303,84 @@ advertises `ServiceLevel` 250. OPC UA clients re-select.
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Bootstrap: the self-form fallback
## Bootstrap: self-first seed ordering
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone. Akka lets only the
**first** listed seed form a *new* cluster; every other node sends `InitJoin` to the seeds and retries —
forever — until one of them answers. A node that boots while its peer is dead therefore never comes Up,
and no downing strategy helps: **auto-down removes the crash outage, not this one.**
Listing both peers in `SeedNodes` does **not** mean either node can cold-start alone — and neither does any
downing strategy: **auto-down removes the crash outage, not this one.** Akka runs a different bootstrap
process depending on whether `seed-nodes[0]` is the node's own address:
`Cluster:SelfFormAfter` closes that gap. `ClusterBootstrapFallback.Arm` (armed for every host built through
`WithOtOpcUaClusterBootstrap`, via an Akka.Hosting `AddStartup` task) waits that long for cluster membership
and, on expiry, calls `Cluster.Join(SelfAddress)` — the node forms a cluster on itself and becomes
operational unattended.
| `seed-nodes[0]` is… | Process Akka runs | Can it form a NEW cluster? |
|---|---|---|
| this node's own address | `FirstSeedNodeProcess` | **Yes** — it `InitJoin`s the other seeds and self-joins once `seed-node-timeout` (5 s) passes with nobody answering. |
| some other node | `JoinSeedNodeProcess` | **No** — it retries `InitJoin` forever, however long the peer stays down. |
| `Cluster:SelfFormAfter` | Behaviour |
|---|---|
| `"00:00:10"` (**default**) | A node with no membership after 10 s self-forms. A live peer answers `InitJoin` in milliseconds, so on any normal boot the fallback never fires. |
| `null` / `≤ 0` | Disabled — the pre-2026-07-22 behaviour: wait on `InitJoin` indefinitely. |
So the ordering *is* the mechanism. **Every node that is a seed of its own mesh lists ITSELF first and its
partner second** (decision 2026-07-22):
**Two guards keep the fallback from creating the split it exists to avoid.**
```jsonc
// central-1 // central-2
"SeedNodes": [ "SeedNodes": [
"akka.tcp://otopcua@central-1:4053", "akka.tcp://otopcua@central-2:4053",
"akka.tcp://otopcua@central-2:4053" "akka.tcp://otopcua@central-1:4053"
] ]
```
**1 — Reachability.** An expired window is not evidence the peer is down, so before self-forming the
fallback TCP-probes the other seed addresses; if any accepts a connection it waits another window instead.
This was added after the live gate below caught the fallback islanding a node: a node bounced by a manual
failover restarted, received `InitJoinAck` from its live peer — a join in flight and healthy — but did not
get the Welcome inside the window, because the peer's ring still held the node's previous incarnation
(`Exiting` → `Down` → `Removed`). The fallback fired on the timer and the node formed a **second cluster**.
The original design assumed `Cluster.Join(SelfAddress)` would be ignored mid-handshake; **it is not — it
wins.** Since manual failover deliberately produces exactly that restart, without this guard every manual
failover could island the node it bounced. Pinned by
`SelfFormBootstrapTests.Reachable_peer_suppresses_self_forming_until_it_goes_away`, whose "peer" is a bare
`TcpListener` that speaks no Akka: reachable, never answering a join.
The rule is **conditional**: it binds only when a node's own address appears in its own seed list. A
driver-only site node is seeded solely by `central-1` (today's docker-dev topology) — it is legitimately
not a seed of anything and is exempt. An unconditional "self must be first" rule would refuse to boot every
site node in the fleet.
**2 — Seed membership.** The fallback fires **only when this node's own address
appears in its own `SeedNodes`**. A node that is not one of its own seeds is never legitimately first, and
if it self-formed, a later-booting real seed would form a second cluster the two could never merge. That is
exactly the current docker-dev topology: the site-a/site-b driver nodes list only `central-1` as a seed, so
they deliberately keep waiting rather than islanding themselves. Under the future per-cluster mesh
(`docs/plans/2026-07-21-per-cluster-mesh-design.md`) each pair node *is* a seed of its own two-node mesh, so
the fallback covers both sides of every pair and the site-node exception disappears.
`AkkaClusterOptionsValidator` enforces it at startup (`AddValidatedOptions` → `ValidateOnStart` in
`AddOtOpcUaCluster`), because the invariant otherwise fails **silently**: the process runs, the port
listens, the node simply never becomes a member. Identity is compared on `Cluster:PublicHostname` (falling
back to `Cluster:Hostname` when blank) **and** `Cluster:Port` — the address Akka puts in `SelfAddress`.
Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it is `0.0.0.0`, which matches
no seed URI anywhere, so the rule would silently exempt the entire rig.
Sequential recovery is island-free by construction: Akka's join protocol prefers an existing cluster (a
booting node only self-forms when **no** seed answered), so a peer booting after the survivor self-formed
simply joins it as the youngest member. The residual risk is both pair nodes cold-starting inside the
window while mutually unreachable — both self-form, which is the same dual-active class the `auto-down`
strategy already accepts, with the same recovery (restart one side).
Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both
nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time
**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the
`auto-down` strategy already accepts, with the same recovery (restart one side).
Pinned by `SelfFormBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real hosts
through the production bootstrap: a lone non-first seed comes Up; a disabled window keeps waiting; a node
absent from its own seed list never self-forms; a node whose peer is merely *reachable* keeps waiting. Every
negative case carries a positive control (an explicit self-join, or dropping the peer listener) so none can
pass merely because the node was unformable.
Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
in-process clusters through the production bootstrap at production failure-detection timings: a lone
cold-start with a dead peer forms alone; a restarted node rejoins its live peer; two nodes cold-starting
together converge on one cluster. Its **falsifiability control** — the old peer-first ordering, same
scenario, never comes Up — is what keeps the other three from going vacuous, and carries a positive-control
self-join proving the node was formable all along. `AkkaClusterOptionsValidatorTests` covers the validator,
including the site-node exemption and the `0.0.0.0` identity trap.
> **Live gate: PASSED (docker-dev, 2026-07-22).** Drilled on the six-node rig, defect first and fix
> second. On the pre-change image, `central-2` started alone (central-1 stopped) and logged **zero** join
> events in 70 s — the `InitJoin` loop, observed rather than argued. On the rebuilt image the same node
> logged the self-form warning **10 s** after start and `Leader is moving node [central-2] to [Up]`, then
> came up with all four admin singletons. A `site-a-2` recreated while central-1 was still down logged
> *"self-form fallback inactive: this node … is not in its own seed list"* and stayed out — the island
> guard on the real topology. Restarting `central-1` produced `Welcome from [central-2]`: it joined the
> self-formed cluster instead of starting a second one, and the four site nodes then joined through it —
> six members, **one** cluster.
### Why the self-form watchdog was retired (2026-07-22)
The first fix for this gap was `Cluster:SelfFormAfter` + `ClusterBootstrapFallback`: wait 10 s for
membership, then call `Cluster.Join(SelfAddress)`. **Both are deleted.** A timer sits *outside* Akka's join
handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join is in flight" —
and `Cluster.Join(SelfAddress)` is **not** ignored mid-handshake: it wins.
> **Live-gate finding (docker-dev, 2026-07-22 — preserved because it is the evidence).** The watchdog
> islanded a node that manual failover had bounced. `central-1` restarted, received `InitJoinAck` from
> `central-2` at 10:49:05 — the join was in flight and healthy — but no Welcome arrived inside the window,
> because the peer's ring still held its previous incarnation (`Exiting` → `Down` → `Removed`, retired at
> 10:49:06). At 10:49:15 the watchdog fired and the node formed a **second cluster**, islanded until an
> operator restarted it. Manual failover deliberately produces exactly that restart, so the two halves of
> that plan collided. A TCP reachability guard (`ea45ace1`) patched that one shape — never self-form while
> a seed peer accepts a connection — and the earlier live gate for the watchdog itself had genuinely
> passed: `central-2` cold-started alone in 10 s and a non-seed `site-a-2` correctly stayed out.
The reachability guard closed the observed failure, not the class: a peer can be TCP-reachable and not
answering, or answering and slow, and the timer cannot tell. Self-first ordering has no such race because
the decision is made *inside* the handshake, by Akka, on the actual `InitJoin` replies. It was also inert
for every non-seed node, so after this change it could never usefully fire. `SelfFormBootstrapTests` is
replaced by `SelfFirstSeedBootstrapTests`, whose `Restarting_node_rejoins_its_live_peer_instead_of_islanding`
is the drilled scenario above.
> **Live gate outstanding for the new mechanism.** Self-first ordering is verified here by in-process
> clusters at production timings, and on the sister project (ScadaBridge `4a6341d8`) on real clusters *and*
> live docker containers. It has **not** been re-drilled on the OtOpcUa docker-dev rig since the swap — the
> `central-2` cold-start-alone drill should be repeated on the rebuilt image. Owned by
> `docs/plans/2026-07-22-per-cluster-mesh-program.md` Phase 7 alongside the auto-down 1-vs-1 gate.
## Primary data-plane gate (writes, acks, alerts emit)