fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)

Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.

Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.

Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
  ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
  process exits (supervisor restarts it) instead of idling with a dead actor system.
  Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
  both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
  fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
  requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
  non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
  ANY single loss needs 3+ members.

Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).

src/Server/ZB.MOM.WW.OtOpcUa.Host/ActorSystemTerminationWatchdog.cs

@@ -0,0 +1,80 @@
+using Akka.Actor;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Logging;
+
+namespace ZB.MOM.WW.OtOpcUa.Host;
+
+/// <summary>
+///     Down-if-alone recovery watchdog (arch-review #459). The 2-node <c>keep-oldest</c> split-brain
+///     resolver downs the <b>lone survivor</b> when the OLDEST node crashes (the survivor is "the side
+///     without the oldest" → <c>DownReachable</c>), and <c>run-coordinated-shutdown-when-down = on</c>
+///     terminates that node's <see cref="ActorSystem"/>. Recovery is by <b>exit-and-rejoin</b>: the
+///     service supervisor (Windows <c>sc.exe failure … restart</c> / docker <c>restart: unless-stopped</c>)
+///     restarts the exited node and it re-forms / rejoins as a fresh incarnation.
+///
+///     <para>This watchdog closes the gap where the <see cref="ActorSystem"/> terminates but the .NET
+///     host process keeps running with a dead actor system (idling forever, never restarted). It watches
+///     <see cref="ActorSystem.WhenTerminated"/>; if the system terminates <b>outside</b> a normal host
+///     shutdown, it stops the application so the supervisor can restart the process. Mirrors the sister
+///     ScadaBridge project's proven pattern.</para>
+///
+///     <para>Registered <b>after</b> <c>AddAkka</c> so it starts after Akka's own hosted service has
+///     built the system; the <see cref="ActorSystem"/> is resolved lazily in <see cref="StartAsync"/>
+///     (never at construction) so it can't race Akka startup. A graceful host stop is distinguished from
+///     an unexpected SBR self-down via <see cref="_stopRequested"/> plus
+///     <see cref="IHostApplicationLifetime.ApplicationStopping"/>, so normal shutdown never logs a false
+///     alarm or double-triggers <see cref="IHostApplicationLifetime.StopApplication"/>.</para>
+/// </summary>
+public sealed class ActorSystemTerminationWatchdog : IHostedService
+{
+    private readonly Func<ActorSystem> _actorSystemAccessor;
+    private readonly IHostApplicationLifetime _lifetime;
+    private readonly ILogger<ActorSystemTerminationWatchdog> _logger;
+    private volatile bool _stopRequested;
+
+    /// <summary>Constructs the watchdog over a lazy <see cref="ActorSystem"/> accessor and the host lifetime.</summary>
+    /// <param name="actorSystemAccessor">Lazy accessor (resolved in <see cref="StartAsync"/>, never at construction, so it can't race Akka startup).</param>
+    /// <param name="lifetime">Host application lifetime used to stop the process on an unexpected self-down.</param>
+    /// <param name="logger">Logger for the critical self-down diagnostic.</param>
+    public ActorSystemTerminationWatchdog(
+        Func<ActorSystem> actorSystemAccessor,
+        IHostApplicationLifetime lifetime,
+        ILogger<ActorSystemTerminationWatchdog> logger)
+    {
+        _actorSystemAccessor = actorSystemAccessor;
+        _lifetime = lifetime;
+        _logger = logger;
+    }
+
+    /// <summary>Wires the <see cref="ActorSystem.WhenTerminated"/> continuation that exits the host on an unexpected self-down.</summary>
+    /// <param name="cancellationToken">Unused; the watchdog only registers a continuation.</param>
+    /// <returns>A completed task.</returns>
+    public Task StartAsync(CancellationToken cancellationToken)
+    {
+        var system = _actorSystemAccessor();
+        system.WhenTerminated.ContinueWith(
+            _ =>
+            {
+                // Expected shutdown: our StopAsync ran, or the host is already stopping. Stay quiet.
+                if (_stopRequested || _lifetime.ApplicationStopping.IsCancellationRequested)
+                    return;
+
+                _logger.LogCritical(
+                    "ActorSystem terminated outside host shutdown (SBR self-down / "
+                    + "run-coordinated-shutdown-when-down). Stopping the host so the service supervisor "
+                    + "restarts this node as a fresh incarnation (2-node keep-oldest exit-and-rejoin recovery).");
+                _lifetime.StopApplication();
+            },
+            TaskContinuationOptions.ExecuteSynchronously);
+        return Task.CompletedTask;
+    }
+
+    /// <summary>Marks a graceful shutdown so the termination continuation stays silent and does not re-trigger stop.</summary>
+    /// <param name="cancellationToken">Unused.</param>
+    /// <returns>A completed task.</returns>
+    public Task StopAsync(CancellationToken cancellationToken)
+    {
+        _stopRequested = true;
+        return Task.CompletedTask;
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs

@@ -296,6 +296,16 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
         ab.WithOtOpcUaRuntimeActors();
 });
 
+// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
+// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
+// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor
+// (sc.exe failure / docker restart: unless-stopped) restarts this node — the 2-node keep-oldest
+// exit-and-rejoin recovery path.
+builder.Services.AddHostedService(sp => new ActorSystemTerminationWatchdog(
+    () => sp.GetRequiredService<ActorSystem>(),
+    sp.GetRequiredService<IHostApplicationLifetime>(),
+    sp.GetRequiredService<ILogger<ActorSystemTerminationWatchdog>>()));
+
 if (hasAdmin)
 {
     // Auth + AdminUI surface only mounted on admin-role nodes. Driver-only nodes have no UI.