feat(cluster): leader-auto-down decision logging + cold-start/flapping/partition/switch-off scenarios (ScadaBridge#106)
Every downing decision of LeaderAutoDowningProvider is now logged under a [leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable), unreachable observed + its window (Warning), skipped because already Down/Exiting, pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own AutoDown logged only the final Down. Decision semantics are unchanged. New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests), built from production BuildHocon with self-first seeds: - both nodes cold-start together -> one cluster, nobody downed past FD + window; - lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up, then the peer cold-starts and joins; - flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the window, repeated past it in total -> never downed; positive control asserts the cuts really produced unreachability; - partition held past the window -> each side downs the other, both stay Up and leader (the accepted dual-active trade); - switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider, only leader-auto-down loads the in-repo provider. Unit: flapping never downs; a flapping member that finally stays unreachable is downed exactly once a full window after its LAST unreachable; each decision log line. Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping test fail; DOWNING log dropped -> log test fails; provider never downs -> partition test fails; auto-down miswired to the new provider -> switch-off theory fails. Docker failover drill not run (it would rebuild the shared scadabridge:latest image and reuse the shared rig volumes while another agent works in the repo); filed as deploy gate ScadaBridge#116. Default remains auto-down. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -102,7 +102,7 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user