Joseph Doherty
d1ade5653b
A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:
- Flip ~380 stale task statuses to completed across March/May/June/July
trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
(premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
-> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
#28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
requirements-traceability 'Pending' clarified as frozen plan-generation
status.
Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.
114 lines
6.7 KiB
Markdown
114 lines
6.7 KiB
Markdown
# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
|
||
|
||
**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
|
||
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
|
||
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
|
||
|
||
## The decision
|
||
|
||
All two-node ScadaBridge clusters (central and every site pair) switch their downing
|
||
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
|
||
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
|
||
|
||
- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
|
||
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
|
||
|
||
The leader among the **reachable** members downs the unreachable peer after the
|
||
stability window. Consequence: a hard crash of **either** node — the active/oldest
|
||
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
|
||
with no operator action and no victim restart required.
|
||
|
||
**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
|
||
each site with no Kubernetes and no SQL available site-side, and "network partitions are
|
||
less of a risk than if this stops working." Availability wins.
|
||
|
||
## The accepted trade (read this before debugging a dual-active)
|
||
|
||
In a **real network partition** (both nodes alive, link cut) each side downs the other
|
||
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
|
||
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
|
||
resolver would instead have sacrificed the younger side. Recovery from dual-active is
|
||
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
|
||
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
|
||
their own — the mutual downing quarantines the association.)
|
||
|
||
## Why the crashed-oldest direction was unsurvivable before (evidence)
|
||
|
||
Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
|
||
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
|
||
|
||
```
|
||
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
|
||
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
|
||
[1] unreachable of [2] members
|
||
```
|
||
|
||
The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
|
||
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
|
||
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
|
||
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
|
||
|
||
```csharp
|
||
// oldest is on the OTHER (unreachable) side:
|
||
if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2
|
||
return DownUnreachable.Instance;
|
||
return DownReachable.Instance; // 1-vs-1 → down MYSELF
|
||
```
|
||
|
||
`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
|
||
oldest side ("the node on the other side is no better" — upstream comment). So two-node
|
||
keep-oldest can never survive an oldest crash. This corrected an earlier
|
||
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
|
||
|
||
## Alternatives rejected
|
||
|
||
| Option | Why not |
|
||
|---|---|
|
||
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
|
||
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
|
||
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
|
||
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
|
||
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
|
||
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
|
||
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
|
||
|
||
## What changed (implementation slice, same session)
|
||
|
||
- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
|
||
`DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
|
||
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
|
||
scoped to keep-oldest.
|
||
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
|
||
provider + `auto-down-unreachable-after` vs the SBR block).
|
||
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
|
||
`deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
|
||
**Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
|
||
together.**
|
||
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
|
||
the victim is down (previously it asserted the outage).
|
||
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
|
||
real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
|
||
YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
|
||
`AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
|
||
a `strategy` parameter (default `auto-down`).
|
||
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
|
||
`docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
|
||
resolved.
|
||
|
||
## Residual operational notes
|
||
|
||
- ~~**Seed-node bootstrap constraint still applies to boot-alone**~~ — **SUPERSEDED
|
||
(2026-07-22), no longer a residual.** As written: only the first seed may self-form a
|
||
cluster, so a node that must BOOT alone while its peer is dead (cold start of only the
|
||
non-first-seed VM, or the survivor crashing while the peer is still down) waits in
|
||
`InitJoin` for its peer. This was **closed** by the self-first seed-node ordering change
|
||
(`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`): every node now lists
|
||
ITSELF first in `seed-nodes`, so any node can cold-start alone and become operational
|
||
unattended, and `StartupValidator` fails the boot if the ordering is broken — regression
|
||
coverage in `SelfFirstSeedBootstrapTests`. The residual cost of self-first-on-both (a
|
||
*truly simultaneous* cold start racing `FirstSeedNodeProcess` on both nodes) is in turn
|
||
covered by the opt-in `ScadaBridge:Cluster:BootstrapGuard` (Gitea #33, default OFF).
|
||
- Monitoring already surfaces dual-active if it ever happens: both nodes report
|
||
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
|
||
two Primaries.
|