Merge branch 'feat/116-leader-auto-down-default': leader-auto-down is the shipped default (#116)

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
Joseph Doherty
2026-11-15 19:57:45 -05:00
36 changed files with 153 additions and 83 deletions
+2 -1
View File
@@ -200,7 +200,8 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); DEV (`vd03`, single node — not production) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
### Cluster & Failover
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix. **It also fixes the freeze / healed-partition dual-active (ScadaBridge#132):** a node its peer has shut out (downed + quarantined) restarts itself; when both sides are shut out the higher address restarts after `ScadaBridge:Cluster:LeaderAutoDown:ShutOutGrace` (60s). Opt-out `LeaderAutoDown:RestartWhenShutOut=false`. Any unplanned ActorSystem termination now exits the Host with code 75 (Windows SCM recovery with `failureflag 1` ignores exit 0).
- **Downing default is `leader-auto-down` since 2026-11-15 (ScadaBridge#116)** — the in-repo `LeaderAutoDowningProvider`, with the same availability-first semantics as the `auto-down` decision below plus the #132 shut-out restart. It was drilled on the rig, docker-env2 and QA (central + site, kill and stop-and-hold). DEV, QA and both rigs set it explicitly. `auto-down` (`docker/docker-compose.auto-down.yml` on the rigs) stays supported for comparison and rollback.
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety; the default until 2026-11-15).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement, now the default (ScadaBridge#106/#116):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). The Akka-bump drill rule (register row 70) is closed because the default no longer depends on `AutoDowning`; re-run the drill if `auto-down` is ever selected again. Its decisions log under the `[leader-auto-down]` prefix. **It also fixes the freeze / healed-partition dual-active (ScadaBridge#132):** a node its peer has shut out (downed + quarantined) restarts itself; when both sides are shut out the higher address restarts after `ScadaBridge:Cluster:LeaderAutoDown:ShutOutGrace` (60s). Opt-out `LeaderAutoDown:RestartWhenShutOut=false`. Any unplanned ActorSystem termination now exits the Host with code 75 (Windows SCM recovery with `failureflag 1` ignores exit 0).
- Both nodes are seed nodes. `min-nr-of-members = 1`.
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
- CoordinatedShutdown for graceful singleton handover.
+2 -2
View File
@@ -18,7 +18,7 @@ This repository is the full **implementation** project for ScadaBridge — the C
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
| Hosting | Windows Server, Windows Service |
| Cluster | Akka.NET Cluster (active/standby, `auto-down` downing) |
| Cluster | Akka.NET Cluster (active/standby, `leader-auto-down` downing) |
| Logging | Serilog (structured) |
### Scale
@@ -88,7 +88,7 @@ Both stacks share the infrastructure services in [`infra/`](infra/) (MS SQL, LDA
| 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. |
| 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
| 12 | Site Event Logging | [docs/requirements/Component-SiteEventLogging.md](docs/requirements/Component-SiteEventLogging.md) | SQLite storage, 30-day retention + 1GB cap, daily purge, paginated remote queries with keyword search. |
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, `auto-down` downing strategy (2026-07-21 availability-over-partition-safety decision; `keep-oldest` stays a supported option), min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/downing/CoordinatedShutdown wiring lives in the Host. |
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, `leader-auto-down` downing strategy (2026-07-21 availability-over-partition-safety decision; `keep-oldest` stays a supported option), min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/downing/CoordinatedShutdown wiring lives in the Host. |
| 14 | Inbound API | [docs/requirements/Component-InboundAPI.md](docs/requirements/Component-InboundAPI.md) | POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object/List), script-based implementations, failures-only logging. |
| 15 | Host | [docs/requirements/Component-Host.md](docs/requirements/Component-Host.md) | Single deployable binary, role-based component registration, per-component config binding (Options pattern), readiness gating, dead letter monitoring, Akka.NET bootstrap, ASP.NET Core hosting for central. |
| 16 | Commons | [docs/requirements/Component-Commons.md](docs/requirements/Component-Commons.md) | Namespace/folder convention (Types/Interfaces/Entities/Messages), shared data types, POCOs, repository interfaces, message contracts with additive-only versioning, UTC timestamp convention, the unified read-only alarm condition model (`AlarmConditionState`/`AlarmKind`), and native alarm source entities + the `IAlarmSubscribableConnection` capability seam. |
+8 -6
View File
@@ -129,18 +129,20 @@ Both UIs are now reachable:
run it, so an env2 build never retags the `scadabridge:latest` the primary rig runs. Unset
keeps the old shared-image behaviour.
Since 2026-11-12 env2 is the soak environment for the `leader-auto-down` rollout and runs
`scadabridge:env2-116` with the opt-in overlay
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml). Keep both
when recreating nodes, and switch both pairs together:
env2 was the soak environment for the `leader-auto-down` rollout from 2026-11-12. Since
2026-11-15 `leader-auto-down` is the shipped default, and env2's mounted appsettings say so, so
plain `deploy.sh` now runs it. The old overlay
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml) is a no-op. For a
comparison run on Akka's `AutoDowning`, use
[`docker-compose.auto-down.yml`](docker-compose.auto-down.yml), switching both pairs together:
```bash
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
```
Plain `deploy.sh` goes back to `scadabridge:latest` on the default `auto-down`. Drill:
Drill:
`DRILL_ENV=env2 DRILL_PAIR=central|site-x bash docker/failover-drill.sh`. Record:
[`docs/plans/2026-11-12-leader-auto-down-env2-drill.md`](../docs/plans/2026-11-12-leader-auto-down-env2-drill.md).
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+25
View File
@@ -0,0 +1,25 @@
# Opt-in overlay: run the central pair and the site-x pair on Akka's deprecated
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
# drills only. Env wins over the mounted appsettings.
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
#
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
#
# Back to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
central-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-x-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-x-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
@@ -1,18 +1,10 @@
# Opt-in overlay: run env2's central pair and its site-x pair on the in-repo
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
# instead of the shipped default `auto-down`. env2 is the soak environment for the
# #116 rollout; mirrors docker/docker-compose.leader-auto-down.yml.
#
# The shipped default is NOT changed by this file: the mounted appsettings still say
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
# for a comparison, use docker-env2/docker-compose.auto-down.yml instead.
#
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
#
# Revert to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+4 -2
View File
@@ -400,14 +400,16 @@ DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh # freeze ins
The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node.
**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time:
**Downing provider (ScadaBridge#106/#116).** Since 2026-11-15 the shipped default is `leader-auto-down`, the in-repo `LeaderAutoDowningProvider`, and every node's mounted appsettings say so. It was drilled on this rig, on docker-env2 and on the QA pair before the flip. To compare against Akka's deprecated `AutoDowning` provider (the default from 2026-07-21 until 2026-11-15), the opt-in overlay `docker/docker-compose.auto-down.yml` switches the central and site-a pairs back through an environment override. Switch both pairs together, never one node at a time:
```bash
docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
# back to the default: the same command without the overlay file
```
`docker/docker-compose.leader-auto-down.yml` is now a no-op, kept because the #116 drill records reference it.
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-a:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+25
View File
@@ -0,0 +1,25 @@
# Opt-in overlay: run the central pair and the site-a pair on Akka's deprecated
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
# drills only. Env wins over the mounted appsettings.
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
#
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
#
# Back to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
central-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-a-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-a-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
+4 -11
View File
@@ -1,17 +1,10 @@
# Opt-in overlay: run the central pair and the site-a pair on the in-repo
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
# instead of the shipped default `auto-down`. Used by the #116 rig failover drill.
#
# The shipped default is NOT changed by this file: the mounted appsettings still say
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
# for a comparison, use docker/docker-compose.auto-down.yml instead.
#
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
#
# Revert to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
+2 -2
View File
@@ -2,8 +2,8 @@
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster runs an availability-first
# downing strategy — `auto-down` (default) or the in-repo `leader-auto-down`
# (ScadaBridge#106; opt-in overlay docker/docker-compose.leader-auto-down.yml) —
# downing strategy — the in-repo `leader-auto-down` (default since 2026-11-15,
# ScadaBridge#106/#116) or Akka's `auto-down` (opt-in overlay docker/docker-compose.auto-down.yml) —
# under which the leader among the REACHABLE members downs the unreachable peer
# after StableAfter, so a hard crash of EITHER node of a pair — the active/oldest
# included — fails over to the survivor. The accepted trade (made explicitly by
+1 -1
View File
@@ -22,7 +22,7 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -22,7 +22,7 @@
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -101,7 +101,7 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
### Downing strategy (auto-down — availability-first)
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
**Since 2026-11-15 (ScadaBridge#116) the default strategy is `leader-auto-down`**, the in-repo `LeaderAutoDowningProvider`. It has the same semantics as the `auto-down` decision below without Akka's deprecated provider, and adds the #132 shut-out restart; see `docs/requirements/Component-ClusterInfrastructure.md`. **Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy was **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
+1 -1
View File
@@ -184,7 +184,7 @@ Options are bound via the .NET Options pattern (`IOptions<T>`). Each component o
| Key | Default | Description |
|-----|---------|-------------|
| `SeedNodes` | — | List of Akka seed-node URIs (`akka.tcp://scadabridge@host:port`). At least 2 required. Must reference remoting ports, not gRPC ports. |
| `SplitBrainResolverStrategy` | `auto-down` | Downing strategy: `"auto-down"` (default) or `"keep-oldest"`. |
| `SplitBrainResolverStrategy` | `leader-auto-down` | Downing strategy: `"leader-auto-down"` (default since 2026-11-15, ScadaBridge#116 — the in-repo `LeaderAutoDowningProvider`), `"auto-down"` (Akka's deprecated `AutoDowning`; the default until 2026-11-15) or `"keep-oldest"`. |
| `StableAfter` | `"00:00:15"` | Sustained unreachability before downing (`auto-down-unreachable-after` under `auto-down`). |
| `HeartbeatInterval` | `"00:00:02"` | Akka failure-detector heartbeat cadence. |
| `FailureDetectionThreshold` | `"00:00:10"` | Acceptable heartbeat pause before a node is considered unreachable. |
+1 -1
View File
@@ -38,7 +38,7 @@ The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGat
### Automatic failover
When the active central node goes down, the Akka cluster's `auto-down` downing strategy (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
When the active central node goes down, the Akka cluster's `leader-auto-down` downing strategy (formerly `auto-down`) (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
## Architecture
+2 -2
View File
@@ -8,7 +8,7 @@
1. Active central node becomes unreachable (process crash, network failure, hardware failure).
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
3. `leader-auto-down` (the default since 2026-11-15, ScadaBridge#116; same rule as the former default `auto-down`): the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability. The survivor logs `[leader-auto-down] DOWNING <address>`.
4. Standby node is promoted to active. Total time: ~25 seconds.
5. Cluster singletons migrate to the new active node.
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
@@ -82,7 +82,7 @@
**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".
**How the system handles it:**
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
- `leader-auto-down` (default since 2026-11-15, ScadaBridge#116) and `auto-down` (decision 2026-07-21; the default until then): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active (accepted trade for ~25 s failover on either-node crash). Under `leader-auto-down` the #132 shut-out restart then restarts the stale side when the partition heals (exit 75, the service manager restarts it); under `auto-down` an operator restarts one side.
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
- Stable-after (15s): the window of sustained unreachability before the downing decision.
@@ -69,7 +69,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
| 70 | **→ #51** **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
| 70 | **CLOSED 2026-11-15 → #116** (was → #51; `leader-auto-down` is now the default, so no deployed tier depends on `AutoDowning`) **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
| 71 | ~~**Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)*~~ **RESOLVED 2026-08-16 (`8b761841`).** One narrowly-keyed Serilog `Filter.ByExcluding` in `LoggerConfigurationFactory`: `Level == Error` AND the `DefaultHealthCheckService` category AND `HealthCheckName == "active-node"` — check-NAME keying is what preserves the constraint (`localdb`/`akka-cluster`/`database`/`required-singletons` keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: `MinimumLevel.Override` can only raise a floor (can't rewrite Error→Information), and reporting `Degraded` would change the HTTP status `/health/active` publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes `/health/active` every 5 s) — covered by the same filter. Fail-before: filter disabled → `StandbyActiveNodeVerdict_IsNotLogged` fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's `/health/active`), and CLAUDE.md's "site nodes map no `/health/*`" claim was stale — corrected. | Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
| 72 | ~~**Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 78 — one fix, the DM instance sweep).** Design decision made and recorded (`docs/plans/2026-08-15-instance-sweep-self-heal-design.md`): **converge, don't merely warn** — the sweep re-applies the stored config through **`HandleDeploy`**, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to `Self`, never `NoSender` (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two `deployment` site events naming both revisions. Safety property beyond the brief: **one repair attempt per (instance, revision) pair**, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → `Sweep_RestartsInstance_ServingAStaleRevision` fails showing rev-1's value served while the store holds rev-2. | Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
| 73 | ~~**`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 80).** New tri-state health-report field **`StartupReconcilerActive`** (`bool?`, matching the `LocalDbReplicationConnected` nullable precedent; additive proto **field 29**, vendored C# regenerated via `docker/regen-proto.sh centralcontrol` with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised **Warning → Error** (no configuration legitimately omits the fetcher on a site node) plus a best-effort `deployment`/`Error` site event so evidence outlives the boot log. Central treatment: a **"No self-heal" Warning badge**, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit `false` is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. | Row-67 fix session | Closed 2026-08-16. | — |
@@ -29,7 +29,7 @@ realised across two projects:
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
configures the downing strategy (`auto-down` by default; `leader-auto-down`, the in-repo port, and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
configures the downing strategy (`leader-auto-down`, the in-repo port, by default since 2026-11-15; Akka's `auto-down` and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
@@ -92,7 +92,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
- Health reporting resumes from the new active node.
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
## Downing Strategy (auto-down — availability-first)
## Downing Strategy (availability-first: `leader-auto-down`, formerly `auto-down`)
**Since 2026-11-15 (ScadaBridge#116) the shipped default is `leader-auto-down`** (next section). It has the same availability-first semantics as the `auto-down` decision below, but uses the in-repo provider instead of Akka's deprecated one. Every deployed tier (DEV vd03, QA vq03/vq04) and both docker rigs set it explicitly. `auto-down` stays a supported value for comparison and rollback.
**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
@@ -100,9 +102,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
### `leader-auto-down` — the in-repo port (ScadaBridge#106; the default since 2026-11-15, #116)
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. It became the default on 2026-11-15, after it passed the docker rig and docker-env2 drills and the QA pair's central and site kill and stop-and-hold drills. On QA, `[leader-auto-down] DOWNING` fired at +26–27 s on a held-down node, and a killed-and-restarted node failed over in about 10 s. Switching an existing pair is an appsettings flip on both nodes together, and rollback is flipping back to `auto-down`. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
@@ -52,19 +52,20 @@ public class ClusterOptions
/// <summary>
/// Downing strategy for unreachable members. Three supported values:
/// <list type="bullet">
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
/// <item><c>leader-auto-down</c> (<b>default</b> since 2026-11-15, ScadaBridge#106/#116) — the
/// in-repo <c>LeaderAutoDowningProvider</c> (Host): the same availability-first semantics as
/// <c>auto-down</c> below, without Akka's deprecated <c>AutoDowning</c>, so it survives that
/// provider's removal from a future Akka.NET and does not log the boot-time deprecation warning.
/// Adds the #132 shut-out restart (<see cref="LeaderAutoDown"/>). Drilled on the docker rig,
/// docker-env2 and the QA pair (central + site, kill and stop-and-hold) before becoming the
/// default.</item>
/// <item><c>auto-down</c> (the default from 2026-07-21 until 2026-11-15; still supported) — availability-first: each
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
/// a true network partition produces two live one-node clusters (dual-active) until
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
/// than a rare partition.</item>
/// <item><c>leader-auto-down</c> (opt-in, ScadaBridge#106) — identical semantics to
/// <c>auto-down</c>, implemented by the in-repo <c>LeaderAutoDowningProvider</c> (Host) instead
/// of Akka's deprecated <c>AutoDowning</c>, so it survives that provider's removal from a
/// future Akka.NET and does not log the boot-time deprecation warning. Not yet the default:
/// the docker failover drill must pass on it first
/// (<c>docs/plans/2026-11-09-autodowning-replacement-plan.md</c>).</item>
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
@@ -76,7 +77,7 @@ public class ClusterOptions
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
/// lowest-address node.
/// </summary>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
public string SplitBrainResolverStrategy { get; set; } = "leader-auto-down";
/// <summary>
/// Time the cluster membership must remain stable before the split-brain
@@ -14,14 +14,16 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
{
/// <summary>
/// Downing strategies supported for ScadaBridge's two-node clusters.
/// <c>auto-down</c> (default) survives a crash of either node at the accepted cost
/// <c>leader-auto-down</c> (default since 2026-11-15, ScadaBridge#116) and <c>auto-down</c>
/// survive a crash of either node at the accepted cost
/// of dual-active during a real partition; <c>keep-oldest</c> is partition-safe but
/// cannot survive a crash of the oldest node. Quorum strategies are rejected:
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
/// fatal crash to the lowest-address node instead of the oldest.
/// <c>leader-auto-down</c> (ScadaBridge#106) is the in-repo port of <c>auto-down</c>
/// (same semantics, no dependency on Akka's deprecated <c>AutoDowning</c>).
/// (same semantics, no dependency on Akka's deprecated <c>AutoDowning</c>), with the #132
/// shut-out restart.
/// </summary>
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
{
@@ -303,12 +303,12 @@ public class AkkaHostedService : IHostedLifecycleService
/// whitespace cannot corrupt the document or be silently misparsed.
///
/// The downing block branches on <see cref="ClusterOptions.SplitBrainResolverStrategy"/>:
/// <c>auto-down</c> (default; decision 2026-07-21) installs Akka's
/// <c>auto-down</c> (the default 2026-07-21 to 2026-11-15) installs Akka's
/// <c>AutoDowning</c> provider with <c>auto-down-unreachable-after</c> =
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
/// downs the unreachable peer, so a crash of either node (oldest included) fails
/// over to the survivor; the accepted trade is dual-active during a real network
/// partition. <c>leader-auto-down</c> (ScadaBridge#106, opt-in) installs the in-repo
/// partition. <c>leader-auto-down</c> (ScadaBridge#106; the default since 2026-11-15, #116) installs the in-repo
/// <see cref="LeaderAutoDowningProvider"/> — the same semantics without Akka's deprecated
/// provider — with its duration at
/// <see cref="LeaderAutoDowningProvider.UnreachableAfterPath"/>.
@@ -357,14 +357,14 @@ public class AkkaHostedService : IHostedLifecycleService
var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
// auto-down (default): AutoDowning provider — the leader among the reachable
// auto-down (the default until 2026-11-15; still supported): AutoDowning provider — the leader among the reachable
// members downs the unreachable peer after StableAfter, so a crash of EITHER
// node fails over to the survivor (dual-active during a real partition is the
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
// the configured active-strategy (keep-oldest), which is partition-safe but
// cannot survive a crash of the oldest node in a two-node cluster.
//
// leader-auto-down (ScadaBridge#106, opt-in): the in-repo LeaderAutoDowningProvider — a
// leader-auto-down (ScadaBridge#106; the default since 2026-11-15, #116): the in-repo LeaderAutoDowningProvider — a
// port of AutoDowning with identical semantics that survives AutoDowning's eventual
// removal from Akka.NET. Its duration lives under our own key, so no
// auto-down-unreachable-after is emitted (which also silences Akka's deprecation warning).
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@localhost:8081",
"akka.tcp://scadabridge@localhost:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -16,7 +16,7 @@
"akka.tcp://scadabridge@localhost:8082",
"akka.tcp://scadabridge@localhost:8085"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -11,10 +11,11 @@ public class ClusterOptionsTests
{
var options = new ClusterOptions();
// 'auto-down' is the default posture (decision 2026-07-21): a crash of either
// node fails over to the survivor; dual-active during a real partition is the
// accepted trade for pairs with no shared lease infrastructure.
Assert.Equal("auto-down", options.SplitBrainResolverStrategy);
// 'leader-auto-down' is the default since 2026-11-15 (ScadaBridge#116): the in-repo port
// of the availability-first 'auto-down' posture (decision 2026-07-21), without Akka's
// deprecated AutoDowning provider. A crash of either node fails over to the survivor.
Assert.Equal("leader-auto-down", options.SplitBrainResolverStrategy);
Assert.True(options.LeaderAutoDown.RestartWhenShutOut);
Assert.Equal(TimeSpan.FromSeconds(15), options.StableAfter);
Assert.Equal(TimeSpan.FromSeconds(2), options.HeartbeatInterval);
Assert.Equal(TimeSpan.FromSeconds(10), options.FailureDetectionThreshold);
@@ -364,4 +364,27 @@ public class HoconBuilderTests
Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
}
/// <summary>
/// ScadaBridge#116: an environment that sets no strategy gets <c>leader-auto-down</c> with
/// <c>LeaderAutoDown</c> at its defaults — the in-repo provider, the StableAfter window under its
/// own key, the shut-out restart ON, and no deprecated <c>auto-down-unreachable-after</c> key.
/// </summary>
[Fact]
public void BuildHocon_UnsetStrategy_DefaultsToLeaderAutoDown()
{
var cluster = new ClusterOptions { SeedNodes = DefaultCluster().SeedNodes };
var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
DefaultNode(), cluster, new[] { "Central" },
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
Assert.Equal(LeaderAutoDowningProvider.ProviderClassName,
config.GetString("akka.cluster.downing-provider-class"));
Assert.Equal(TimeSpan.FromSeconds(15), config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath));
Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
Assert.False(config.HasPath("akka.cluster.auto-down-unreachable-after"));
Assert.False(config.HasPath("akka.cluster.split-brain-resolver.active-strategy"));
}
}
@@ -20,7 +20,7 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// members (<c>KeepOldest.OldestDecision</c>: <c>otherSide == 1 &amp;&amp; thisSide &gt;= 2</c>).
/// With a two-node cluster the younger survivor therefore takes <c>DownReachable</c> — it downs
/// ITSELF — so only a YOUNGER-node crash is survivable.</item>
/// <item><c>auto-down</c> (production default, decision 2026-07-21) — the leader among
/// <item><c>auto-down</c> (the production default from 2026-07-21 until 2026-11-15; leader-auto-down since, #116) — the leader among
/// the reachable members downs the unreachable peer after the stability window, so a
/// crash of EITHER node fails over to the survivor; the accepted trade is dual-active
/// during a real network partition.</item>
@@ -42,7 +42,7 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
string role = "Central", TimeSpan? stableAfter = null,
int? portA = null, int? portB = null,
TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null,
string strategy = "auto-down")
string strategy = "leader-auto-down")
{
var f = new TwoNodeClusterFixture();
f.PortA = portA ?? GetFreeTcpPort();
@@ -56,8 +56,9 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
}
/// <summary>Starts a node from production HOCON; used by StartAsync and by restart-scenarios.
/// <paramref name="strategy"/> defaults to the production posture (auto-down, decision
/// 2026-07-21); pass "keep-oldest" to exercise the legacy SBR path.</summary>
/// <paramref name="strategy"/> defaults to the production posture (leader-auto-down, the default
/// since 2026-11-15, ScadaBridge#116); pass "auto-down" for Akka's AutoDowning or "keep-oldest"
/// to exercise the legacy SBR path.</summary>
/// <param name="port">The remoting port for the node.</param>
/// <param name="role">The Akka role assigned to the node.</param>
/// <param name="stableAfter">How long membership must be stable before downing decisions; defaults to a scaled test value.</param>
@@ -67,7 +68,7 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
/// <returns>The started actor system.</returns>
public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null,
TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null,
string strategy = "auto-down")
string strategy = "leader-auto-down")
{
var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port };
var clusterOptions = new ClusterOptions