fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459) #463

2026-07-15T10:53:12-04:00

dohertj2 commented

2026-07-15 10:53:12 -04:00

Corrects #459. 2-node keep-oldest recovery works fine (ScadaBridge parity); OtOpcUa was missing the supervision pieces + the docs claimed in-place oldest-crash failover.

ActorSystemTerminationWatchdog (Host): exits the process on an unexpected ActorSystem self-down so the supervisor restarts it. 3 unit tests.
docker-dev: restart: unless-stopped + both peers in SeedNodes.
docs/Redundancy.md: rewrote the recovery section (younger-loss = in-place; oldest-loss = exit-and-rejoin under supervision); flagged HardKillFailoverTests as non-representative.

Cluster.Tests 29/29, watchdog 3/3, solution builds. Live watchdog-image re-verify pending (host docker disk full). Refs #459.

Corrects #459. 2-node keep-oldest recovery works fine (ScadaBridge parity); OtOpcUa was missing the supervision pieces + the docs claimed in-place oldest-crash failover. - ActorSystemTerminationWatchdog (Host): exits the process on an unexpected ActorSystem self-down so the supervisor restarts it. 3 unit tests. - docker-dev: restart: unless-stopped + both peers in SeedNodes. - docs/Redundancy.md: rewrote the recovery section (younger-loss = in-place; oldest-loss = exit-and-rejoin under supervision); flagged HardKillFailoverTests as non-representative. Cluster.Tests 29/29, watchdog 3/3, solution builds. Live watchdog-image re-verify pending (host docker disk full). Refs #459.

dohertj2 added 1 commit 2026-07-15 10:53:13 -04:00

fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459 )

v2-ci / unit-tests (pull_request) Failing after 12m9s

Details

v2-ci / build (pull_request) Successful in 5m12s

Details

b0eb653bad

Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.

Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.

Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
  ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
  process exits (supervisor restarts it) instead of idling with a dead actor system.
  Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
  both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
  fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
  requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
  non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
  ANY single loss needs 3+ members.

Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).

dohertj2 closed this pull request

2026-07-15 10:53:16 -04:00

dohertj2 deleted branch fix/459-sbr-recovery-scadabridge-parity

2026-07-15 10:53:16 -04:00

v2-ci / unit-tests (pull_request) Failing after 12m9s

Details

v2-ci / build (pull_request) Successful in 5m12s

Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#463