dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog

Browse Source 
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
Joseph Doherty
2026-07-22 07:42:06 -04:00
parent a78425ea8f
commit 3f24d4d6bf
9 changed files with 625 additions and 506 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker-dev/docker-compose.yml
+20 -7
View File
@@ -13,7 +13,9 @@
# central-1, central-2 OTOPCUA_ROLES=admin,driver — the ONLY UI + deploy
# singleton, plus the MAIN cluster's OPC UA publishers.
# Reachable at http://localhost:9200 (via Traefik).
# central-1 is the Akka seed node; central-2 joins it.
# Both are Akka seed nodes, each listing ITSELF first
# (2026-07-22) so either can cold-start while the other
# is down; the site nodes seed off central-1 only.
# site-a-1, site-a-2 OTOPCUA_ROLES=driver — driver-only members of the same
# site-b-1, site-b-2 mesh, scoped to SITE-A / SITE-B by ClusterId. They
# serve no UI and authenticate no users; the central
@@ -136,7 +138,8 @@ services:
# ── Central cluster (2-node fused admin+driver) ─────────────────────────────
# The only UI + deploy singleton; also the MAIN cluster's OPC UA publishers.
# central-1 seeds the single Akka mesh that every other node joins.
# central-1 seeds the single Akka mesh that every other node joins; central-2 is a seed too
# (listing itself first), so it can form the mesh alone if central-1 is down.
central-1: &otopcua-host
build:
@@ -176,8 +179,13 @@ services:
Cluster__Port: "4053"
Cluster__PublicHostname: "central-1"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
# the 2-node keep-oldest exit-and-rejoin recovery path.
# re-join the mesh via EITHER peer, not only central-1.
#
# ORDER IS LOAD-BEARING (2026-07-22): each node lists ITSELF first. Akka runs
# FirstSeedNodeProcess — the only path that can form a NEW cluster when no peer answers
# InitJoin — exclusively for seed-nodes[0]; every other node retries InitJoin forever. So a
# node listing its partner first cannot cold-start while that partner is down. Enforced at
# boot by AkkaClusterOptionsValidator; see docs/Redundancy.md.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
Cluster__Roles__0: "admin"
@@ -248,9 +256,11 @@ services:
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "central-2"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1. SELF FIRST:
# central-2 lists itself as seed-nodes[0], which is what lets it cold-start while central-1
# is down (it previously listed central-1 first and simply never came Up in that case).
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -314,6 +324,9 @@ services:
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "site-a-1"
# Site nodes are deliberately NOT seeds — they join the central pair's mesh. The self-first
# seed rule is conditional for exactly this reason (it binds only when a node's own address
# is in its own seed list), so these configs are exempt rather than broken.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__Roles__0: "driver"
<<: *secrets-env