fix(redundancy): 2-node SBR exit-and-rejoin recovery — watchdog + restart supervision + both-node seeds (#459)

Corrects the #459 finding. 2-node keep-oldest recovery works fine (the ScadaBridge
sister project proves it); OtOpcUa was missing the supervision pieces that make it
automatic, and docs/Redundancy.md wrongly claimed in-place oldest-crash failover.

Mechanism (confirmed on a 2-container rig + by decompiling Akka KeepOldest.OldestDecision):
on an OLDEST-node crash keep-oldest downs the LONE survivor (DownReachable including
myself) — down-if-alone can't rescue a lone survivor (its branch needs >=2 survivors).
Recovery is exit-and-rejoin: run-coordinated-shutdown-when-down terminates the node and
the service supervisor restarts it. My earlier 'total outage' was a docker-dev artifact
(no restart policy); production Install-Services.ps1 already has sc.exe failure restart.

Changes (ScadaBridge parity):
- ActorSystemTerminationWatchdog (Host, registered after AddAkka): watches
  ActorSystem.WhenTerminated and on an unexpected self-down calls StopApplication so the
  process exits (supervisor restarts it) instead of idling with a dead actor system.
  Distinguishes graceful shutdown via _stopRequested + ApplicationStopping. 3 unit tests.
- docker-dev: restart: unless-stopped on the host anchor (models production supervision) +
  both redundancy peers in SeedNodes so a restarted node re-forms via either peer.
- docs/Redundancy.md: rewrote the split-brain recovery section — younger-loss = in-place
  fast failover; oldest-loss = exit-and-rejoin under supervision (not in-place); the three
  requirements (supervisor + watchdog + both-node seeds); flagged HardKillFailoverTests as
  non-representative (Transport.Shutdown, not a real crash). Instant in-place takeover on
  ANY single loss needs 3+ members.

Cluster.Tests 29/29 (SBR guards), watchdog tests 3/3, full solution builds.
Live re-verify of the watchdog image pending (host docker disk full).

docs/Redundancy.md

@@ -161,23 +161,46 @@ provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of
 `ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
 (`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
 the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
-the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and both the cluster singletons and
-the `driver` role-leader fail over. (Only an *explicit* `NoDowning`, e.g.
+the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
+recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
 `akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
 indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
 `keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
 must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
 express it).
 
-`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest
-member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within
-~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum`
-are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition
-re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node
-triggers the same failover — verified by `HardKillFailoverTests` (its negative control confirms failover
-survives removing the typed option, and only breaks under an explicit `NoDowning`).
+`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
+are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
+this is inherent to any 2-node cluster:
 
-There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
+| Node lost | What keep-oldest does | Recovery |
+|---|---|---|
+| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
+| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
+
+**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
+sister project runs the same pattern):
+
+1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
+   `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
+   `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
+2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
+   watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
+   `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
+   idling forever with a dead actor system.
+3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
+
+> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
+> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
+> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
+> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
+> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
+> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
+> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
+
+There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
+do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
+or a lightweight witness) — a deliberate future option, not the current 2-node posture.
 
 ## Primary data-plane gate (writes, acks, alerts emit)