fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge sister project proves it); OtOpcUa was missing the supervision pieces that make it automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover. Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision): on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors). Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact (no restart policy); production Install-Services.ps1 already has sc.exe failure restart. Changes (ScadaBridge parity): - ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the process exits (supervisor restarts it) instead of idling with a dead actor system. Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests. - docker-dev: restart: unless-stopped on the host anchor (models production supervision) + both redundancy peers in SeedNodes so a restarted node re-forms via either peer. - docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on ANY single loss needs 3+ members. Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds. Live re-verify of the watchdog image pending (host docker disk full).
This commit is contained in:
@@ -133,6 +133,14 @@ services:
|
||||
# host raise the VM memory or run fewer host services.
|
||||
mem_limit: 2g
|
||||
mem_reservation: 1g
|
||||
# Restart supervision. A 2-node keep-oldest SBR downs the LONE survivor when the
|
||||
# OLDEST node crashes (the survivor is "the side without the oldest" → DownReachable
|
||||
# → run-coordinated-shutdown-when-down terminates it). Recovery is by exit-and-rejoin:
|
||||
# the supervisor restarts the exited node and it re-forms/rejoins as a fresh
|
||||
# incarnation (the ScadaBridge pattern). Without a restart policy the exited node
|
||||
# stays down and the cluster looks like a total outage. Inherited by every host node
|
||||
# via the anchor merge.
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
sql: { condition: service_healthy }
|
||||
migrator: { condition: service_completed_successfully }
|
||||
@@ -143,7 +151,11 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-1"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) so a restarted node can
|
||||
# re-join the mesh via EITHER peer, not only central-1. With restart supervision this is
|
||||
# the 2-node keep-oldest exit-and-rejoin recovery path.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
Cluster__Roles__1: "driver"
|
||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||
@@ -204,7 +216,9 @@ services:
|
||||
Cluster__Hostname: "0.0.0.0"
|
||||
Cluster__Port: "4053"
|
||||
Cluster__PublicHostname: "central-2"
|
||||
# Both redundancy peers are seeds (#459 / ScadaBridge parity) — see central-1.
|
||||
Cluster__SeedNodes__0: "akka.tcp://otopcua@central-1:4053"
|
||||
Cluster__SeedNodes__1: "akka.tcp://otopcua@central-2:4053"
|
||||
Cluster__Roles__0: "admin"
|
||||
Cluster__Roles__1: "driver"
|
||||
Security__Jwt__SigningKey: "docker-dev-signing-key-with-at-least-32-bytes-of-utf8-content-12345"
|
||||
|
||||
Reference in New Issue
Block a user