feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)
SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair (central + site, kill and stop-and-hold) before the flip. - ClusterOptions default + validator/HOCON comments updated. - Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/ node appsettings now say leader-auto-down. The leader-auto-down overlays are kept as no-ops (drill records reference them); new docker-compose.auto-down.yml overlays keep Akka AutoDowning available for comparison drills. - Tests: default asserts leader-auto-down; new HOCON test for an unset strategy; TwoNodeClusterFixture defaults to the production posture. - Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster, README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed. No deploy needed: DEV, QA and both rigs set the key explicitly. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -101,7 +101,7 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
### Downing strategy (auto-down — availability-first)
|
||||
|
||||
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
**Since 2026-11-15 (ScadaBridge#116) the default strategy is `leader-auto-down`**, the in-repo `LeaderAutoDowningProvider`. It has the same semantics as the `auto-down` decision below without Akka's deprecated provider, and adds the #132 shut-out restart; see `docs/requirements/Component-ClusterInfrastructure.md`. **Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy was **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
|
||||
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
|
||||
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||
|
||||
@@ -184,7 +184,7 @@ Options are bound via the .NET Options pattern (`IOptions<T>`). Each component o
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `SeedNodes` | — | List of Akka seed-node URIs (`akka.tcp://scadabridge@host:port`). At least 2 required. Must reference remoting ports, not gRPC ports. |
|
||||
| `SplitBrainResolverStrategy` | `auto-down` | Downing strategy: `"auto-down"` (default) or `"keep-oldest"`. |
|
||||
| `SplitBrainResolverStrategy` | `leader-auto-down` | Downing strategy: `"leader-auto-down"` (default since 2026-11-15, ScadaBridge#116 — the in-repo `LeaderAutoDowningProvider`), `"auto-down"` (Akka's deprecated `AutoDowning`; the default until 2026-11-15) or `"keep-oldest"`. |
|
||||
| `StableAfter` | `"00:00:15"` | Sustained unreachability before downing (`auto-down-unreachable-after` under `auto-down`). |
|
||||
| `HeartbeatInterval` | `"00:00:02"` | Akka failure-detector heartbeat cadence. |
|
||||
| `FailureDetectionThreshold` | `"00:00:10"` | Acceptable heartbeat pause before a node is considered unreachable. |
|
||||
|
||||
@@ -38,7 +38,7 @@ The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGat
|
||||
|
||||
### Automatic failover
|
||||
|
||||
When the active central node goes down, the Akka cluster's `auto-down` downing strategy (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
When the active central node goes down, the Akka cluster's `leader-auto-down` downing strategy (formerly `auto-down`) (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
1. Active central node becomes unreachable (process crash, network failure, hardware failure).
|
||||
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
|
||||
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
|
||||
3. `leader-auto-down` (the default since 2026-11-15, ScadaBridge#116; same rule as the former default `auto-down`): the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability. The survivor logs `[leader-auto-down] DOWNING <address>`.
|
||||
4. Standby node is promoted to active. Total time: ~25 seconds.
|
||||
5. Cluster singletons migrate to the new active node.
|
||||
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
|
||||
@@ -82,7 +82,7 @@
|
||||
**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".
|
||||
|
||||
**How the system handles it:**
|
||||
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
|
||||
- `leader-auto-down` (default since 2026-11-15, ScadaBridge#116) and `auto-down` (decision 2026-07-21; the default until then): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active (accepted trade for ~25 s failover on either-node crash). Under `leader-auto-down` the #132 shut-out restart then restarts the stale side when the partition heals (exit 75, the service manager restarts it); under `auto-down` an operator restarts one side.
|
||||
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
|
||||
- Stable-after (15s): the window of sustained unreachability before the downing decision.
|
||||
|
||||
|
||||
@@ -69,7 +69,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
|
||||
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
|
||||
| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
|
||||
| 70 | **→ #51** **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
|
||||
| 70 | **CLOSED 2026-11-15 → #116** (was → #51; `leader-auto-down` is now the default, so no deployed tier depends on `AutoDowning`) **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
|
||||
| 71 | ~~**Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)*~~ **RESOLVED 2026-08-16 (`8b761841`).** One narrowly-keyed Serilog `Filter.ByExcluding` in `LoggerConfigurationFactory`: `Level == Error` AND the `DefaultHealthCheckService` category AND `HealthCheckName == "active-node"` — check-NAME keying is what preserves the constraint (`localdb`/`akka-cluster`/`database`/`required-singletons` keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: `MinimumLevel.Override` can only raise a floor (can't rewrite Error→Information), and reporting `Degraded` would change the HTTP status `/health/active` publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes `/health/active` every 5 s) — covered by the same filter. Fail-before: filter disabled → `StandbyActiveNodeVerdict_IsNotLogged` fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's `/health/active`), and CLAUDE.md's "site nodes map no `/health/*`" claim was stale — corrected. | Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
|
||||
| 72 | ~~**Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 78 — one fix, the DM instance sweep).** Design decision made and recorded (`docs/plans/2026-08-15-instance-sweep-self-heal-design.md`): **converge, don't merely warn** — the sweep re-applies the stored config through **`HandleDeploy`**, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to `Self`, never `NoSender` (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two `deployment` site events naming both revisions. Safety property beyond the brief: **one repair attempt per (instance, revision) pair**, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → `Sweep_RestartsInstance_ServingAStaleRevision` fails showing rev-1's value served while the store holds rev-2. | Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
|
||||
| 73 | ~~**`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 80).** New tri-state health-report field **`StartupReconcilerActive`** (`bool?`, matching the `LocalDbReplicationConnected` nullable precedent; additive proto **field 29**, vendored C# regenerated via `docker/regen-proto.sh centralcontrol` with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised **Warning → Error** (no configuration legitimately omits the fetcher on a site node) plus a best-effort `deployment`/`Error` site event so evidence outlives the boot log. Central treatment: a **"No self-heal" Warning badge**, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit `false` is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. | Row-67 fix session | Closed 2026-08-16. | — |
|
||||
|
||||
@@ -29,7 +29,7 @@ realised across two projects:
|
||||
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
|
||||
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
|
||||
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
|
||||
configures the downing strategy (`auto-down` by default; `leader-auto-down`, the in-repo port, and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
|
||||
configures the downing strategy (`leader-auto-down`, the in-repo port, by default since 2026-11-15; Akka's `auto-down` and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
|
||||
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
|
||||
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
|
||||
|
||||
@@ -92,7 +92,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
- Health reporting resumes from the new active node.
|
||||
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
|
||||
|
||||
## Downing Strategy (auto-down — availability-first)
|
||||
## Downing Strategy (availability-first: `leader-auto-down`, formerly `auto-down`)
|
||||
|
||||
**Since 2026-11-15 (ScadaBridge#116) the shipped default is `leader-auto-down`** (next section). It has the same availability-first semantics as the `auto-down` decision below, but uses the in-repo provider instead of Akka's deprecated one. Every deployed tier (DEV vd03, QA vq03/vq04) and both docker rigs set it explicitly. `auto-down` stays a supported value for comparison and rollback.
|
||||
|
||||
**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
|
||||
|
||||
@@ -100,9 +102,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
|
||||
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106; the default since 2026-11-15, #116)
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. It became the default on 2026-11-15, after it passed the docker rig and docker-env2 drills and the QA pair's central and site kill and stop-and-hold drills. On QA, `[leader-auto-down] DOWNING` fired at +26–27 s on a held-down node, and a killed-and-restarted node failed over in about 10 s. Switching an existing pair is an appsettings flip on both nodes together, and rollback is flipping back to `auto-down`. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user