fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)
v2-ci / unit-tests (pull_request) Failing after 12m9s
v2-ci / build (pull_request) Successful in 5m12s

Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.

Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.

Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
  ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
  process exits (supervisor restarts it) instead of idling with a dead actor system.
  Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
  both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
  fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
  requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
  non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
  ANY single loss needs 3+ members.

Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).
This commit is contained in:
Joseph Doherty
2026-07-15 10:52:36 -04:00
parent b980eeb148
commit b0eb653bad
5 changed files with 229 additions and 10 deletions
+14
View File
@@ -133,6 +133,14 @@ services:
# host raise the VM memory or run fewer host services.
mem_limit: 2g
mem_reservation: 1g
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
# stays down and the cluster looks like a total outage. Inherited by every host node
# via the anchor merge.
restart: unless-stopped
depends_on:
sql: { condition: service_healthy }
migrator: { condition: service_completed_successfully }
@@ -143,7 +151,11 @@ services:
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "central-1"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
# the 2-node keep-oldest exit-and-rejoin recovery path.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
@@ -204,7 +216,9 @@ services:
Cluster__Hostname: "0.0.0.0"
Cluster__Port: "4053"
Cluster__PublicHostname: "central-2"
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
Cluster__Roles__0: "admin"
Cluster__Roles__1: "driver"
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"