feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)

Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.

docs/plans/2026-07-08-deferred-work-register.md

@@ -48,5 +48,5 @@ Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md
 
 | # | Item | Where noted | Rationale for deferral | Revisit trigger |
 |---|------|-------------|------------------------|-----------------|
-| SBR | **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition *without* the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until the crashed node restarts; only a younger-node crash fails over. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` | Remedy is a production SBR topology/strategy decision (keep-majority + a 3rd/lighthouse seed node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. | Before the next production deployment that adds a central node, or the first real active-node crash. |
+| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
 | vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |

docs/plans/2026-07-21-auto-down-availability-decision.md

@@ -0,0 +1,108 @@
+# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
+
+**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
+deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
+`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
+
+## The decision
+
+All two-node ScadaBridge clusters (central and every site pair) switch their downing
+strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
+(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
+
+- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
+- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
+
+The leader among the **reachable** members downs the unreachable peer after the
+stability window. Consequence: a hard crash of **either** node — the active/oldest
+included — fails over to the survivor in ~25s (10s failure detection + 15s window),
+with no operator action and no victim restart required.
+
+**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
+each site with no Kubernetes and no SQL available site-side, and "network partitions are
+less of a risk than if this stops working." Availability wins.
+
+## The accepted trade (read this before debugging a dual-active)
+
+In a **real network partition** (both nodes alive, link cut) each side downs the other
+and continues as a one-node cluster: **both run active** — two oldest-Up members, two
+sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
+resolver would instead have sacrificed the younger side. Recovery from dual-active is
+operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
+its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
+their own — the mutual downing quarantines the association.)
+
+## Why the crashed-oldest direction was unsurvivable before (evidence)
+
+Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
+(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
+
+```
+SBR took decision Akka.Cluster.SBR.DownReachable and is downing
+[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
+[1] unreachable of [2] members
+```
+
+The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
+restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
+victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
+(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
+
+```csharp
+// oldest is on the OTHER (unreachable) side:
+if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
+    return DownUnreachable.Instance;
+return DownReachable.Instance;                         // 1-vs-1 → down MYSELF
+```
+
+`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
+oldest side ("the node on the other side is no better" — upstream comment). So two-node
+keep-oldest can never survive an oldest crash. This corrected an earlier
+mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
+
+## Alternatives rejected
+
+| Option | Why not |
+|---|---|
+| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
+| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
+| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
+| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
+| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
+| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
+| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
+
+## What changed (implementation slice, same session)
+
+- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
+  `DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
+- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
+  scoped to keep-oldest.
+- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
+  provider + `auto-down-unreachable-after` vs the SBR block).
+- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
+  `deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
+  **Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
+  together.**
+- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
+  the victim is down (previously it asserted the outage).
+- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
+  real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
+  YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
+  `AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
+  a `strategy` parameter (default `auto-down`).
+- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
+  `docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
+  resolved.
+
+## Residual operational notes
+
+- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
+  may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
+  restarts), but a node that must BOOT alone while its peer is dead (cold start of only
+  the non-first-seed VM, or the survivor crashing while the peer is still down) still
+  waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
+  self-first seed override).
+- Monitoring already surfaces dual-active if it ever happens: both nodes report
+  `IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
+  two Primaries.

docs/requirements/Component-ClusterInfrastructure.md

@@ -92,27 +92,33 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
 - Health reporting resumes from the new active node.
 - Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
 
-## Split-Brain Resolution
+## Downing Strategy (auto-down — availability-first)
 
-The system uses the Akka.NET **keep-oldest** split-brain resolver strategy:
+**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
 
-- On a network partition, the node that has been in the cluster longest remains active. The younger node downs itself.
-- **Stable-after duration**: 15 seconds. The cluster membership must remain stable (no changes) for 15 seconds before the resolver acts to down unreachable nodes. This prevents premature downing during startup or rolling restarts.
-- **`down-if-alone = on`**: The keep-oldest resolver is configured with `down-if-alone` enabled. If the oldest node finds itself alone (no other reachable members), it downs itself rather than continuing as a single-node cluster. This prevents the oldest node from running in isolation during a network partition while the younger node also forms its own cluster.
-- **Why keep-oldest**: With only two nodes, quorum-based strategies (static-quorum, keep-majority) cannot distinguish "one node crashed" from "network partition" — both sides see fewer than quorum and both would down themselves, resulting in total cluster shutdown. Keep-oldest with `down-if-alone` provides safe singleton ownership — at most one node runs the cluster singleton at any time.
+- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues (as before). If the **active/oldest** node crashes, the younger survivor becomes leader among the reachable members, downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — **no operator action and no victim restart required**. This closes the keep-oldest total-outage gap.
+- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
+- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
 
-### Down-if-alone recovery
+### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
 
-When a node downs itself (via `down-if-alone`, or any other SBR decision), the resolver is configured with `run-coordinated-shutdown-when-down = on`, so the self-down runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
+- **keep-oldest** (used until 2026-07-21): partition-safe, but in a two-node cluster a crash of the **oldest** is a total outage. `KeepOldest.OldestDecision` only lets `down-if-alone` rescue the survivor when the surviving side has **≥ 2 members** (`otherSide == 1 && thisSide >= 2`); with 1-vs-1 the younger survivor takes `DownReachable` — it downs *itself*. Verified live on the docker rig: the survivor logged `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`, exited, and looped on `InitJoin`. The strategy remains supported in `ClusterOptions` (`SplitBrainResolverStrategy: keep-oldest`) for deployments that prefer partition-safety over availability.
+- **static-quorum (quorum-size 1)**: Akka's `IsTooManyMembers` guard (`members > quorum*2-1`, i.e. `2 > 1`) returns **DownAll** on any unreachability — total shutdown, strictly worse.
+- **keep-majority**: a 1-vs-1 split keeps the side with the lowest address, which just moves the fatal crash from "the oldest" to "the lowest-address node".
+- **lease-majority**: needs a shared lease store (Kubernetes API, SQL, …) reachable from both nodes — not available at sites (one node per VM, no shared infrastructure).
 
-1. Self-down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
+### Downed-node recovery
+
+When a node is downed (auto-downed by its peer after a partition heals, or a keep-oldest self-down where that strategy is configured), `run-coordinated-shutdown-when-down = on` runs `CoordinatedShutdown` and **terminates that node's `ActorSystem`**. The Host process must not keep running with a dead actor system — it would serve nothing and be restarted by nobody. The recovery contract is:
+
+1. Down ⇒ `CoordinatedShutdown` ⇒ `ActorSystem` termination.
 2. The Host watches `ActorSystem.WhenTerminated`; a termination that is **not** the host's own `StopAsync` triggers `IHostApplicationLifetime.StopApplication()`, so **the process exits**.
 3. The service supervisor restarts it — docker `restart: unless-stopped`, or Windows service recovery actions (`sc.exe failure … restart/…`).
-4. The restarted process rejoins as a **fresh incarnation** (the keep-oldest resolver handles the rejoin cleanly; there is no stale membership to reconcile) — **but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
+4. The restarted process rejoins as a **fresh incarnation** — **but only while a peer still holding cluster state is reachable**. A lone restarted node that is *not* the first seed cannot re-form a cluster on its own (see the seed-node bootstrap constraint below); it waits for its peer.
 
-**Seed-node bootstrap constraint.** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. This is why the two-node keep-oldest **oldest/active-node crash is a total-outage gap**: after the oldest dies the younger survivor self-downs, and it cannot re-bootstrap alone. Recovery is operator-driven — either restart the dead first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge. Removing the gap itself is the **registered deferred keep-oldest topology/strategy decision** (master tracker 2026-07-08, owner: user).
+**Seed-node bootstrap constraint (still applies).** Only the FIRST seed listed in `Cluster:SeedNodes` may self-join to form a *new* cluster. All nodes list the same first seed (e.g. `scadabridge-central-a`), so a lone restarted non-first-seed node (with the first seed still down) loops on `InitJoin` forever — never `Up`, never routable. Under auto-down this constraint no longer causes the active-crash outage (the survivor keeps running and never restarts), but it still bites when a node must **boot alone** — e.g. a cold start of only the non-first-seed VM, or the survivor itself crashing while its peer is still dead. Recovery is operator-driven — either restart the first-seed node (preferred) or restart the survivor with a self-first seed override (`ScadaBridge__Cluster__SeedNodes__0` = self, `__1` = peer). The repo does not ship self-first ordering per node: with both nodes self-first a simultaneous cold start risks two independent one-node clusters that never merge.
 
-The docker failover drill (`docker/failover-drill.sh`) exercises this per direction: `standby` mode proves SBR downing + singleton continuity on the oldest; `active` mode measures the registered total-outage gap and the recovery-on-restart path.
+The docker failover drill (`docker/failover-drill.sh`) proves both directions: `standby` mode kills the younger node (active untouched, zero routing blips); `active` mode kills the active/oldest node and asserts the survivor **takes over while the victim is still down**.
 
 ## Single-Node Operation