feat(cluster): leader-auto-down decision logging + cold-start/flapping/partition/switch-off scenarios (ScadaBridge#106)

Every downing decision of LeaderAutoDowningProvider is now logged under a
[leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable),
unreachable observed + its window (Warning), skipped because already Down/Exiting,
pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept
pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own
AutoDown logged only the final Down. Decision semantics are unchanged.

New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests),
built from production BuildHocon with self-first seeds:
- both nodes cold-start together -> one cluster, nobody downed past FD + window;
- lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up,
  then the peer cold-starts and joins;
- flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the
  window, repeated past it in total -> never downed; positive control asserts the cuts
  really produced unreachability;
- partition held past the window -> each side downs the other, both stay Up and
  leader (the accepted dual-active trade);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider,
  only leader-auto-down loads the in-repo provider.
Unit: flapping never downs; a flapping member that finally stays unreachable is downed
exactly once a full window after its LAST unreachable; each decision log line.

Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable
ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping
test fail; DOWNING log dropped -> log test fails; provider never downs -> partition
test fails; auto-down miswired to the new provider -> switch-off theory fails.

Docker failover drill not run (it would rebuild the shared scadabridge:latest image
and reuse the shared rig volumes while another agent works in the repo); filed as
deploy gate ScadaBridge#116. Default remains auto-down.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 04:35:43 -05:00
parent a13aabc7c3
commit 456c6659e7
6 changed files with 436 additions and 10 deletions
@@ -8,6 +8,16 @@
`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value.
Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the
docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.
**Update 2026-11-10 (ScadaBridge#106 complete):** every downing decision is now logged with a `[leader-auto-down]`
prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed,
timed out on a non-leader → pending, leadership gained/lost, and `DOWNING` at Warning before `Cluster.Down`). New
real-cluster scenarios in `IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs`: joint cold start, lone cold start
with the peer dead then the peer joining, a flapping peer (Akka `trttl` throttler blackhole, unreachable shorter than the
window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held
past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (`auto-down` →
`AutoDowning`, `keep-oldest` → `SplitBrainResolverProvider`). The docker drill and rollout are tracked as the deploy gate
**ScadaBridge#116**.
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
## 1. Current state (read from the code)
@@ -102,7 +102,7 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)