fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)
Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge sister project proves it); OtOpcUa was missing the supervision pieces that make it automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover. Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision): on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors). Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact (no restart policy); production Install-Services.ps1 already has sc.exe failure restart. Changes (ScadaBridge parity): - ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the process exits (supervisor restarts it) instead of idling with a dead actor system. Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests. - docker-dev: restart: unless-stopped on the host anchor (models production supervision) + both redundancy peers in SeedNodes so a restarted node re-forms via either peer. - docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on ANY single loss needs 3+ members. Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds. Live re-verify of the watchdog image pending (host docker disk full).
This commit is contained in:
@@ -296,6 +296,16 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
|
||||
ab.WithOtOpcUaRuntimeActors();
|
||||
});
|
||||
|
||||
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
|
||||
// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
|
||||
// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor
|
||||
// (sc.exe failure / docker restart: unless-stopped) restarts this node — the 2-node keep-oldest
|
||||
// exit-and-rejoin recovery path.
|
||||
builder.Services.AddHostedService(sp => new ActorSystemTerminationWatchdog(
|
||||
() => sp.GetRequiredService<ActorSystem>(),
|
||||
sp.GetRequiredService<IHostApplicationLifetime>(),
|
||||
sp.GetRequiredService<ILogger<ActorSystemTerminationWatchdog>>()));
|
||||
|
||||
if (hasAdmin)
|
||||
{
|
||||
// Auth + AdminUI surface only mounted on admin-role nodes. Driver-only nodes have no UI.
|
||||
|
||||
Reference in New Issue
Block a user