Implement the in-repo AutoDowning-equivalent DowningProvider (plan in #51) #106

New Issue

2026-11-09T16:25:09-05:00

dohertj2 commented

2026-11-09 16:25:09 -05:00

Implement the AutoDowning replacement planned in #51 (docs/plans/2026-11-09-autodowning-replacement-plan.md): an in-repo DowningProvider that reproduces Akka's deprecated AutoDowning (the leader among reachable members downs a peer unreachable for stable-after), added as a new downing-strategy value alongside auto-down on Akka 1.5.62. Prove it with HOCON/validator unit tests, two-node real-cluster clones of SbrFailoverTests' oldest-crash and younger-crash cases, and DRILL_MODE=active bash docker/failover-drill.sh on the rig; only then switch the default and roll rig → env2 → QA → DEV/sites (rollback = appsettings back to auto-down). Must land before any Akka.NET upgrade that removes AutoDowning.

Implement the AutoDowning replacement planned in #51 (docs/plans/2026-11-09-autodowning-replacement-plan.md): an in-repo `DowningProvider` that reproduces Akka's deprecated AutoDowning (the leader among reachable members downs a peer unreachable for `stable-after`), added as a new downing-strategy value alongside `auto-down` on Akka 1.5.62. Prove it with HOCON/validator unit tests, two-node real-cluster clones of SbrFailoverTests' oldest-crash and younger-crash cases, and `DRILL_MODE=active bash docker/failover-drill.sh` on the rig; only then switch the default and roll rig → env2 → QA → DEV/sites (rollback = appsettings back to auto-down). Must land before any Akka.NET upgrade that removes AutoDowning.

dohertj2 added the type:feature priority:p2 labels 2026-11-09 16:25:09 -05:00

dohertj2 commented

2026-11-09 16:51:25 -05:00

Provider implemented on main ab561faa — default unchanged (auto-down): new strategy leader-auto-down = LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.Cluster 1.5.62 AutoDowning/AutoDown/AutoDownBase (acts only as leader among reachable members; downs a peer unreachable for StableAfter; non-leader keeps pending and downs on becoming leader; reachable/removed peers forgotten; Down/Exiting skipped). Timeout from scadabridge.cluster.leader-auto-down.unreachable-after so the deprecated key (and Akka's boot warning) is never emitted; DownRemovalMargin read like AutoDowning. BuildHocon emits it only for the new value (auto-down / keep-oldest HOCON byte-identical); validator accepts it. Placed in Host, not ClusterInfrastructure (that project is Akka-free by rule) — plan doc updated. Evidence: 13 unit tests ported from Akka's spec + real two-node oldest-crash/younger-crash clones pass with the same outcomes as auto-down; the loaded provider on both nodes is verified to be LeaderAutoDowningProvider; five deliberate breakages each fail tests. ClusterInfrastructure 46, Host 640, IntegrationTests cluster subset 23; re-run green after rebase. Remaining gates before flipping the default (tracked in #51's plan §5–6): docker failover drill (active + standby) on central + a site pair, live boot log without the deprecation warning, partition dual-active check, then rig → env2 → QA → DEV with pairs stopped/started together.

Provider implemented on main ab561faa — **default unchanged (auto-down)**: new strategy `leader-auto-down` = LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.Cluster 1.5.62 AutoDowning/AutoDown/AutoDownBase (acts only as leader among reachable members; downs a peer unreachable for StableAfter; non-leader keeps pending and downs on becoming leader; reachable/removed peers forgotten; Down/Exiting skipped). Timeout from `scadabridge.cluster.leader-auto-down.unreachable-after` so the deprecated key (and Akka's boot warning) is never emitted; DownRemovalMargin read like AutoDowning. BuildHocon emits it only for the new value (auto-down / keep-oldest HOCON byte-identical); validator accepts it. Placed in Host, not ClusterInfrastructure (that project is Akka-free by rule) — plan doc updated. Evidence: 13 unit tests ported from Akka's spec + real two-node oldest-crash/younger-crash clones pass with the same outcomes as auto-down; the loaded provider on both nodes is verified to be LeaderAutoDowningProvider; five deliberate breakages each fail tests. ClusterInfrastructure 46, Host 640, IntegrationTests cluster subset 23; re-run green after rebase. **Remaining gates before flipping the default** (tracked in #51's plan §5–6): docker failover drill (active + standby) on central + a site pair, live boot log without the deprecation warning, partition dual-active check, then rig → env2 → QA → DEV with pairs stopped/started together.

dohertj2 referenced this issue

2026-11-10 04:32:09 -05:00

Deploy gate: leader-auto-down rollout. Rig drill PASSED (2026-11-12); next env2 → QA → DEV #116

dohertj2 commented

2026-11-10 04:45:29 -05:00

Completed on main 456c6659, on top of ab561faa. The default is still auto-down, and the new strategy is opt-in via SplitBrainResolverStrategy: leader-auto-down.

Added

Decision logging. Every downing decision is logged with a [leader-auto-down] prefix:
- the start-up snapshot (leader, window, members already unreachable);
- a peer seen as unreachable, with its window (Warning);
- a peer skipped because it is already Down or Exiting;
- a pending Down cancelled because the peer became reachable again or was removed;
- a timeout on a non-leader, which keeps the Down pending;
- leadership gained or lost;
- DOWNING (Warning), logged before Cluster.Down.
The decision logic itself is unchanged.
IntegrationTests/Cluster/LeaderAutoDownScenarioTests. These run a real two-node cluster built from the production BuildHocon with self-first seeds. All 7 pass:
- both nodes cold-start together → one cluster forms and nobody is downed, checked past failure detection plus the window;
- a lone cold start with the peer dead (the shape where down-if-alone self-downs on a cold recreate) → the node stays Up, then the peer cold-starts and joins;
- a flapping peer, cut off with Akka's trttl throttler Blackhole. Each cut makes it unreachable for less than the window, and the cuts repeat for longer than the window in total. It is never downed. A positive control asserts that at least 3 of the 4 cuts really made the peer unreachable;
- a partition held past the window → each side downs the other and both stay Up as leader (the accepted dual-active, plan §6);
- switch off: auto-down loads AutoDowning, keep-oldest loads SplitBrainResolverProvider, and only leader-auto-down loads the in-repo provider.
Unit tests (Host). A flapping member is never downed. A member that flaps and then stays unreachable is downed exactly once, a full window after its last unreachable event. Each decision log line is asserted with EventFilter.

Mutation-proved. Each of these deliberate breakages makes tests fail:

not cancelling the timer when the peer is reachable again → the flap-then-stay unit test fails;
ignoring ReachableMember → both unit flapping tests, the cancel-log test and the real-cluster flapping test fail;
dropping the DOWNING log → the log test fails;
a provider that never downs → the partition test fails;
auto-down wired to the new provider → the switch-off theory fails.

Verification. dotnet build ZB.MOM.WW.ScadaBridge.slnx has 0 warnings. Host.Tests 679/679, ClusterInfrastructure.Tests 56/56, IntegrationTests .Cluster. subset 30/30. After rebasing, the LeaderAutoDown and HoconBuilder tests were re-run and passed (34 and 10).

Not done: the docker failover drill. Running it would rebuild the shared scadabridge:latest image and reuse the shared rig volumes while another agent was working in the repo. The drill (central + site-a, active and standby modes, boot-log check) and the rollout from rig → env2 → QA → DEV are tracked as deploy gate #116. Closing this issue as implemented.

Completed on main `456c6659`, on top of `ab561faa`. The default is still **`auto-down`**, and the new strategy is opt-in via `SplitBrainResolverStrategy: leader-auto-down`. **Added** - **Decision logging.** Every downing decision is logged with a `[leader-auto-down]` prefix: - the start-up snapshot (leader, window, members already unreachable); - a peer seen as unreachable, with its window (Warning); - a peer skipped because it is already Down or Exiting; - a pending Down cancelled because the peer became reachable again or was removed; - a timeout on a non-leader, which keeps the Down pending; - leadership gained or lost; - `DOWNING` (Warning), logged before `Cluster.Down`. The decision logic itself is unchanged. - **`IntegrationTests/Cluster/LeaderAutoDownScenarioTests`.** These run a real two-node cluster built from the production `BuildHocon` with self-first seeds. All 7 pass: - both nodes cold-start together → one cluster forms and nobody is downed, checked past failure detection plus the window; - a lone cold start with the peer dead (the shape where `down-if-alone` self-downs on a cold recreate) → the node stays Up, then the peer cold-starts and joins; - a **flapping peer**, cut off with Akka's `trttl` throttler `Blackhole`. Each cut makes it unreachable for less than the window, and the cuts repeat for longer than the window in total. It is never downed. A positive control asserts that at least 3 of the 4 cuts really made the peer unreachable; - a **partition held past the window** → each side downs the other and both stay Up as leader (the accepted dual-active, plan §6); - **switch off**: `auto-down` loads `AutoDowning`, `keep-oldest` loads `SplitBrainResolverProvider`, and only `leader-auto-down` loads the in-repo provider. - **Unit tests (Host).** A flapping member is never downed. A member that flaps and then stays unreachable is downed exactly once, a full window after its *last* unreachable event. Each decision log line is asserted with `EventFilter`. **Mutation-proved.** Each of these deliberate breakages makes tests fail: - not cancelling the timer when the peer is reachable again → the flap-then-stay unit test fails; - ignoring `ReachableMember` → both unit flapping tests, the cancel-log test and the real-cluster flapping test fail; - dropping the DOWNING log → the log test fails; - a provider that never downs → the partition test fails; - `auto-down` wired to the new provider → the switch-off theory fails. **Verification.** `dotnet build ZB.MOM.WW.ScadaBridge.slnx` has 0 warnings. Host.Tests 679/679, ClusterInfrastructure.Tests 56/56, IntegrationTests `.Cluster.` subset 30/30. After rebasing, the LeaderAutoDown and HoconBuilder tests were re-run and passed (34 and 10). **Not done: the docker failover drill.** Running it would rebuild the shared `scadabridge:latest` image and reuse the shared rig volumes while another agent was working in the repo. The drill (central + site-a, active and standby modes, boot-log check) and the rollout from rig → env2 → QA → DEV are tracked as deploy gate **#116**. Closing this issue as implemented.

dohertj2 referenced this issue

2026-11-10 04:45:29 -05:00

Plan a replacement for deprecated Akka AutoDowning before an Akka.NET upgrade removes it #51

dohertj2 closed this issue

2026-11-10 04:45:29 -05:00

dohertj2 referenced this issue from a commit

2026-11-12 20:16:11 -05:00

test(docker): leader-auto-down rig failover drill — #116 rig gate PASS on central + site-a

dohertj2 referenced this issue

2026-11-12 21:38:49 -05:00

Frozen node returns as a permanent second active after being downed (ThisActorSystemQuarantinedEvent ignored) — auto-down and leader-auto-down #132

Sign in to join this conversation.