Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.
Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.
- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
(kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.
Refs #116, #106, #132
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
Joseph Doherty