fix(redundancy): 2-node SBR exit-and-rejoin recovery (#459) #464

2026-07-15T10:53:52-04:00

dohertj2 commented

2026-07-15 10:53:52 -04:00

Corrects #459 — 2-node keep-oldest recovery works via exit-and-rejoin (ScadaBridge parity): watchdog + restart supervision + both-node seeds. Cluster.Tests 29/29, watchdog 3/3. Live watchdog-image re-verify pending (host docker disk full). Refs #459.

dohertj2 added 1 commit 2026-07-15 10:53:53 -04:00

fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459 )

v2-ci / unit-tests (pull_request) Failing after 12m9s

Details

v2-ci / build (pull_request) Successful in 5m12s

Details

b0eb653bad

Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.

Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.

Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
  ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
  process exits (supervisor restarts it) instead of idling with a dead actor system.
  Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
  both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
  fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
  requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
  non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
  ANY single loss needs 3+ members.

Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).

dohertj2 merged commit 57003a23a5 into master

2026-07-15 10:54:06 -04:00

dohertj2 referenced this pull request

2026-07-15 10:54:33 -04:00

[archreview Crit-1/SBR] Non-transparent redundancy: crashing the OLDEST Akka node does not fail over #459

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#464