feat(cluster): leader-auto-down decision logging + cold-start/flapping/partition/switch-off scenarios (ScadaBridge#106)
Every downing decision of LeaderAutoDowningProvider is now logged under a [leader-auto-down] prefix: start-up snapshot (leader, window, already-unreachable), unreachable observed + its window (Warning), skipped because already Down/Exiting, pending Down cancelled on reachable-again/removed, timed out on a non-leader and kept pending, leadership gained/lost, and DOWNING (Warning) before Cluster.Down. Akka's own AutoDown logged only the final Down. Decision semantics are unchanged. New real two-node scenarios (IntegrationTests/Cluster/LeaderAutoDownScenarioTests), built from production BuildHocon with self-first seeds: - both nodes cold-start together -> one cluster, nobody downed past FD + window; - lone cold start with the peer dead (the down-if-alone cold-recreate shape) stays Up, then the peer cold-starts and joins; - flapping peer via Akka's trttl throttler Blackhole: unreachable shorter than the window, repeated past it in total -> never downed; positive control asserts the cuts really produced unreachability; - partition held past the window -> each side downs the other, both stay Up and leader (the accepted dual-active trade); - switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider, only leader-auto-down loads the in-repo provider. Unit: flapping never downs; a flapping member that finally stays unreachable is downed exactly once a full window after its LAST unreachable; each decision log line. Mutation-proved: no timer cancel on reachable -> flap-then-stay unit fails; reachable ignored -> both unit flapping tests, the cancel-log test and the real-cluster flapping test fail; DOWNING log dropped -> log test fails; provider never downs -> partition test fails; auto-down miswired to the new provider -> switch-off theory fails. Docker failover drill not run (it would rebuild the shared scadabridge:latest image and reuse the shared rig volumes while another agent works in the repo); filed as deploy gate ScadaBridge#116. Default remains auto-down. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -8,6 +8,16 @@
|
||||
`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value.
|
||||
Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the
|
||||
docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.
|
||||
|
||||
**Update 2026-11-10 (ScadaBridge#106 complete):** every downing decision is now logged with a `[leader-auto-down]`
|
||||
prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed,
|
||||
timed out on a non-leader → pending, leadership gained/lost, and `DOWNING` at Warning before `Cluster.Down`). New
|
||||
real-cluster scenarios in `IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs`: joint cold start, lone cold start
|
||||
with the peer dead then the peer joining, a flapping peer (Akka `trttl` throttler blackhole, unreachable shorter than the
|
||||
window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held
|
||||
past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (`auto-down` →
|
||||
`AutoDowning`, `keep-oldest` → `SplitBrainResolverProvider`). The docker drill and rollout are tracked as the deploy gate
|
||||
**ScadaBridge#116**.
|
||||
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
|
||||
|
||||
## 1. Current state (read from the code)
|
||||
|
||||
@@ -102,7 +102,7 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
### Why not the alternatives (all verified against Akka.NET 1.5.62 source, 2026-07-21)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user