Merge branch 'feat/116-leader-auto-down-default': leader-auto-down is the shipped default (#116)
Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -200,7 +200,8 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); DEV (`vd03`, single node — not production) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
|
||||
### Cluster & Failover
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix. **It also fixes the freeze / healed-partition dual-active (ScadaBridge#132):** a node its peer has shut out (downed + quarantined) restarts itself; when both sides are shut out the higher address restarts after `ScadaBridge:Cluster:LeaderAutoDown:ShutOutGrace` (60s). Opt-out `LeaderAutoDown:RestartWhenShutOut=false`. Any unplanned ActorSystem termination now exits the Host with code 75 (Windows SCM recovery with `failureflag 1` ignores exit 0).
|
||||
- **Downing default is `leader-auto-down` since 2026-11-15 (ScadaBridge#116)** — the in-repo `LeaderAutoDowningProvider`, with the same availability-first semantics as the `auto-down` decision below plus the #132 shut-out restart. It was drilled on the rig, docker-env2 and QA (central + site, kill and stop-and-hold). DEV, QA and both rigs set it explicitly. `auto-down` (`docker/docker-compose.auto-down.yml` on the rigs) stays supported for comparison and rollback.
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety; the default until 2026-11-15).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement, now the default (ScadaBridge#106/#116):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). The Akka-bump drill rule (register row 70) is closed because the default no longer depends on `AutoDowning`; re-run the drill if `auto-down` is ever selected again. Its decisions log under the `[leader-auto-down]` prefix. **It also fixes the freeze / healed-partition dual-active (ScadaBridge#132):** a node its peer has shut out (downed + quarantined) restarts itself; when both sides are shut out the higher address restarts after `ScadaBridge:Cluster:LeaderAutoDown:ShutOutGrace` (60s). Opt-out `LeaderAutoDown:RestartWhenShutOut=false`. Any unplanned ActorSystem termination now exits the Host with code 75 (Windows SCM recovery with `failureflag 1` ignores exit 0).
|
||||
- Both nodes are seed nodes. `min-nr-of-members = 1`.
|
||||
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
||||
- CoordinatedShutdown for graceful singleton handover.
|
||||
|
||||
@@ -18,7 +18,7 @@ This repository is the full **implementation** project for ScadaBridge — the C
|
||||
| Authentication | Direct LDAP/AD bind (LDAPS/StartTLS), JWT sessions |
|
||||
| Notifications | Delivered from the central cluster (Email via SMTP/OAuth2-M365 or on-prem Exchange EWS; SMS via Twilio REST); store-and-forwarded from sites |
|
||||
| Hosting | Windows Server, Windows Service |
|
||||
| Cluster | Akka.NET Cluster (active/standby, `auto-down` downing) |
|
||||
| Cluster | Akka.NET Cluster (active/standby, `leader-auto-down` downing) |
|
||||
| Logging | Serilog (structured) |
|
||||
|
||||
### Scale
|
||||
@@ -88,7 +88,7 @@ Both stacks share the infrastructure services in [`infra/`](infra/) (MS SQL, LDA
|
||||
| 10 | Security & Auth | [docs/requirements/Component-Security.md](docs/requirements/Component-Security.md) | Direct LDAP bind (LDAPS/StartTLS), JWT sessions (HMAC-SHA256, 15-min refresh, 30-min idle), role-based authorization (incl. the `Operator`/`Verifier` two-person secured-write roles + policies), site-scoped permissions. |
|
||||
| 11 | Health Monitoring | [docs/requirements/Component-HealthMonitoring.md](docs/requirements/Component-HealthMonitoring.md) | 30s report interval, 60s offline threshold, monotonic sequence numbers, raw error counts, tag resolution counts, dead letter monitoring. |
|
||||
| 12 | Site Event Logging | [docs/requirements/Component-SiteEventLogging.md](docs/requirements/Component-SiteEventLogging.md) | SQLite storage, 30-day retention + 1GB cap, daily purge, paginated remote queries with keyword search. |
|
||||
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, `auto-down` downing strategy (2026-07-21 availability-over-partition-safety decision; `keep-oldest` stays a supported option), min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/downing/CoordinatedShutdown wiring lives in the Host. |
|
||||
| 13 | Cluster Infrastructure | [docs/requirements/Component-ClusterInfrastructure.md](docs/requirements/Component-ClusterInfrastructure.md) | Akka.NET cluster, `leader-auto-down` downing strategy (2026-07-21 availability-over-partition-safety decision; `keep-oldest` stays a supported option), min-nr-of-members=1, 2s/10s/15s failure detection, CoordinatedShutdown, automatic dual-node recovery. The `ClusterInfrastructure` project owns the `ClusterOptions` config model; the Akka bootstrap/downing/CoordinatedShutdown wiring lives in the Host. |
|
||||
| 14 | Inbound API | [docs/requirements/Component-InboundAPI.md](docs/requirements/Component-InboundAPI.md) | POST /api/{methodName}, X-API-Key header, flat JSON, extended type system (Object/List), script-based implementations, failures-only logging. |
|
||||
| 15 | Host | [docs/requirements/Component-Host.md](docs/requirements/Component-Host.md) | Single deployable binary, role-based component registration, per-component config binding (Options pattern), readiness gating, dead letter monitoring, Akka.NET bootstrap, ASP.NET Core hosting for central. |
|
||||
| 16 | Commons | [docs/requirements/Component-Commons.md](docs/requirements/Component-Commons.md) | Namespace/folder convention (Types/Interfaces/Entities/Messages), shared data types, POCOs, repository interfaces, message contracts with additive-only versioning, UTC timestamp convention, the unified read-only alarm condition model (`AlarmConditionState`/`AlarmKind`), and native alarm source entities + the `IAlarmSubscribableConnection` capability seam. |
|
||||
|
||||
@@ -129,18 +129,20 @@ Both UIs are now reachable:
|
||||
run it, so an env2 build never retags the `scadabridge:latest` the primary rig runs. Unset
|
||||
keeps the old shared-image behaviour.
|
||||
|
||||
Since 2026-11-12 env2 is the soak environment for the `leader-auto-down` rollout and runs
|
||||
`scadabridge:env2-116` with the opt-in overlay
|
||||
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml). Keep both
|
||||
when recreating nodes, and switch both pairs together:
|
||||
env2 was the soak environment for the `leader-auto-down` rollout from 2026-11-12. Since
|
||||
2026-11-15 `leader-auto-down` is the shipped default, and env2's mounted appsettings say so, so
|
||||
plain `deploy.sh` now runs it. The old overlay
|
||||
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml) is a no-op. For a
|
||||
comparison run on Akka's `AutoDowning`, use
|
||||
[`docker-compose.auto-down.yml`](docker-compose.auto-down.yml), switching both pairs together:
|
||||
|
||||
```bash
|
||||
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
|
||||
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
|
||||
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
|
||||
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
|
||||
```
|
||||
|
||||
Plain `deploy.sh` goes back to `scadabridge:latest` on the default `auto-down`. Drill:
|
||||
Drill:
|
||||
`DRILL_ENV=env2 DRILL_PAIR=central|site-x bash docker/failover-drill.sh`. Record:
|
||||
[`docs/plans/2026-11-12-leader-auto-down-env2-drill.md`](../docs/plans/2026-11-12-leader-auto-down-env2-drill.md).
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Opt-in overlay: run the central pair and the site-x pair on Akka's deprecated
|
||||
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
|
||||
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
|
||||
# drills only. Env wins over the mounted appsettings.
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
#
|
||||
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
|
||||
#
|
||||
# Back to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
central-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-x-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-x-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
@@ -1,18 +1,10 @@
|
||||
# Opt-in overlay: run env2's central pair and its site-x pair on the in-repo
|
||||
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
|
||||
# instead of the shipped default `auto-down`. env2 is the soak environment for the
|
||||
# #116 rollout; mirrors docker/docker-compose.leader-auto-down.yml.
|
||||
#
|
||||
# The shipped default is NOT changed by this file: the mounted appsettings still say
|
||||
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
|
||||
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
|
||||
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
|
||||
# for a comparison, use docker-env2/docker-compose.auto-down.yml instead.
|
||||
#
|
||||
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
|
||||
#
|
||||
# Revert to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
+4
-2
@@ -400,14 +400,16 @@ DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh # freeze ins
|
||||
|
||||
The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node.
|
||||
|
||||
**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time:
|
||||
**Downing provider (ScadaBridge#106/#116).** Since 2026-11-15 the shipped default is `leader-auto-down`, the in-repo `LeaderAutoDowningProvider`, and every node's mounted appsettings say so. It was drilled on this rig, on docker-env2 and on the QA pair before the flip. To compare against Akka's deprecated `AutoDowning` provider (the default from 2026-07-21 until 2026-11-15), the opt-in overlay `docker/docker-compose.auto-down.yml` switches the central and site-a pairs back through an environment override. Switch both pairs together, never one node at a time:
|
||||
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
|
||||
docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
|
||||
up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
# back to the default: the same command without the overlay file
|
||||
```
|
||||
|
||||
`docker/docker-compose.leader-auto-down.yml` is now a no-op, kept because the #116 drill records reference it.
|
||||
|
||||
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
|
||||
|
||||
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Opt-in overlay: run the central pair and the site-a pair on Akka's deprecated
|
||||
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
|
||||
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
|
||||
# drills only. Env wins over the mounted appsettings.
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
#
|
||||
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
#
|
||||
# Back to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
central-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-a-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-a-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
@@ -1,17 +1,10 @@
|
||||
# Opt-in overlay: run the central pair and the site-a pair on the in-repo
|
||||
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
|
||||
# instead of the shipped default `auto-down`. Used by the #116 rig failover drill.
|
||||
#
|
||||
# The shipped default is NOT changed by this file: the mounted appsettings still say
|
||||
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
|
||||
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
|
||||
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
|
||||
# for a comparison, use docker/docker-compose.auto-down.yml instead.
|
||||
#
|
||||
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
#
|
||||
# Revert to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
||||
#
|
||||
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster runs an availability-first
|
||||
# downing strategy — `auto-down` (default) or the in-repo `leader-auto-down`
|
||||
# (ScadaBridge#106; opt-in overlay docker/docker-compose.leader-auto-down.yml) —
|
||||
# downing strategy — the in-repo `leader-auto-down` (default since 2026-11-15,
|
||||
# ScadaBridge#106/#116) or Akka's `auto-down` (opt-in overlay docker/docker-compose.auto-down.yml) —
|
||||
# under which the leader among the REACHABLE members downs the unreachable peer
|
||||
# after StableAfter, so a hard crash of EITHER node of a pair — the active/oldest
|
||||
# included — fails over to the survivor. The accepted trade (made explicitly by
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -101,7 +101,7 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
### Downing strategy (auto-down — availability-first)
|
||||
|
||||
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
**Since 2026-11-15 (ScadaBridge#116) the default strategy is `leader-auto-down`**, the in-repo `LeaderAutoDowningProvider`. It has the same semantics as the `auto-down` decision below without Akka's deprecated provider, and adds the #132 shut-out restart; see `docs/requirements/Component-ClusterInfrastructure.md`. **Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy was **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
|
||||
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
|
||||
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||
|
||||
@@ -184,7 +184,7 @@ Options are bound via the .NET Options pattern (`IOptions<T>`). Each component o
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `SeedNodes` | — | List of Akka seed-node URIs (`akka.tcp://scadabridge@host:port`). At least 2 required. Must reference remoting ports, not gRPC ports. |
|
||||
| `SplitBrainResolverStrategy` | `auto-down` | Downing strategy: `"auto-down"` (default) or `"keep-oldest"`. |
|
||||
| `SplitBrainResolverStrategy` | `leader-auto-down` | Downing strategy: `"leader-auto-down"` (default since 2026-11-15, ScadaBridge#116 — the in-repo `LeaderAutoDowningProvider`), `"auto-down"` (Akka's deprecated `AutoDowning`; the default until 2026-11-15) or `"keep-oldest"`. |
|
||||
| `StableAfter` | `"00:00:15"` | Sustained unreachability before downing (`auto-down-unreachable-after` under `auto-down`). |
|
||||
| `HeartbeatInterval` | `"00:00:02"` | Akka failure-detector heartbeat cadence. |
|
||||
| `FailureDetectionThreshold` | `"00:00:10"` | Acceptable heartbeat pause before a node is considered unreachable. |
|
||||
|
||||
@@ -38,7 +38,7 @@ The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGat
|
||||
|
||||
### Automatic failover
|
||||
|
||||
When the active central node goes down, the Akka cluster's `auto-down` downing strategy (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
When the active central node goes down, the Akka cluster's `leader-auto-down` downing strategy (formerly `auto-down`) (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
|
||||
1. Active central node becomes unreachable (process crash, network failure, hardware failure).
|
||||
2. Akka.NET failure detection triggers after ~10 seconds (2s heartbeat, 10s threshold).
|
||||
3. `auto-down`: the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability.
|
||||
3. `leader-auto-down` (the default since 2026-11-15, ScadaBridge#116; same rule as the former default `auto-down`): the leader among the reachable members downs the unreachable node after 15 seconds of sustained unreachability. The survivor logs `[leader-auto-down] DOWNING <address>`.
|
||||
4. Standby node is promoted to active. Total time: ~25 seconds.
|
||||
5. Cluster singletons migrate to the new active node.
|
||||
6. Load balancer detects the failed node via `/health/ready` and routes traffic to the surviving node.
|
||||
@@ -82,7 +82,7 @@
|
||||
**Symptoms:** Both nodes believe they are the active node. Logs show "Cluster partition detected".
|
||||
|
||||
**How the system handles it:**
|
||||
- `auto-down` (decision 2026-07-21): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active until an operator restarts one side (accepted trade for ~25 s failover on either-node crash).
|
||||
- `leader-auto-down` (default since 2026-11-15, ScadaBridge#116) and `auto-down` (decision 2026-07-21; the default until then): each side downs the unreachable peer after 15 s and continues as a one-node cluster, so a real partition leaves BOTH nodes active (accepted trade for ~25 s failover on either-node crash). Under `leader-auto-down` the #132 shut-out restart then restarts the stale side when the partition heals (exit 75, the service manager restarts it); under `auto-down` an operator restarts one side.
|
||||
- `keep-oldest` (opt-in): the older node survives and the younger is downed; but a crash of the oldest is a total outage.
|
||||
- Stable-after (15s): the window of sustained unreachability before the downing decision.
|
||||
|
||||
|
||||
@@ -69,7 +69,7 @@ All 7 fix-now items landed via PLAN-04/05/06/07/08 (verified in review 08 round
|
||||
| 66 | ~~**No counter for import-session cap rejections** *(Low)*~~ **RESOLVED 2026-08-15 (`90ce0999`).** `scadabridge.central.transport.import_sessions.rejected` on the existing allowlisted meter, emitted ONLY at the post-eviction recheck — the cap is tested twice and a store full of expired sessions reclaims and succeeds, so counting the first test would fire routinely on any node whose sweep hadn't run; duplicate-session-id re-`Open` is also excluded. New tests use delta-over-scoped-action with unique per-test keys (the pre-existing gauge tests' absolute asserts are row 74). | Row-53 implementation note | Closed 2026-08-15. | — |
|
||||
| 67 | ~~**Startup reconcile stores a fetched instance config but does not start the InstanceActor** *(Medium-Low, availability)*~~ **RESOLVED 2026-08-15 (`482165cb`).** Root cause: InstanceActors are created from exactly two paths — the boot-time SQLite load in `DeploymentManagerActor.PreStart` and an explicit deploy/enable command — and the reconcile pass fires ~5s AFTER the boot load with a pure file write (`StoreDeployedConfigIfNewerAsync`) as its only side effect. Fix: reconcile collects the enabled instances it stored and Tells a new `EnsureInstanceStartedCommand` to the DM **singleton proxy**, so the actor is created under the singleton no matter which node's pass found the gap. Handler is advisory and idempotent: no-op if already running; re-reads the durable row (authority on config body AND `is_enabled`, so a disable landing after the fetch wins); defers to `_deployWarms`/`_terminatingActorsByName`/`_inFlightDeployments` so deploy-gate ordering and all-or-nothing-per-instance are untouched (it starts an already-persisted config, never applies one); quiet no-op on an absent row (normal on an unreplicated site whose peer healed its own file). The heal is no longer silent — a Warning site event `Instance <X> started by startup reconcile self-heal` fires. Fail-before by behavioral revert: 3 of 8 new tests fail pre-fix, including the exact live symptom (`debug snapshot` → `instanceNotFound`); the 5 negative guards (disabled / failed-fetch / unknown / idempotence) green in both states. SiteRuntime 613/613, Host 504/504. Sibling gaps found during the fix recorded as rows 72/73. | Proven live during the rows-46–53 close-out (2026-08-15) | Closed 2026-08-15. | — |
|
||||
| 69 | ~~**Both site-a nodes' LocalDb files corrupt — replication down, one node crash-looping**~~ **RESOLVED 2026-08-15 (operational recovery, no code change).** Found live during this wave: site-a-b threw `SQLite Error 11: 'database disk image is malformed'` on every sync call (replication `connected: false`, backlog 892, faulted session retried ~every 60s) and, after the redeploy, site-a-a crash-looped (SIGABRT ×12) on the same error at `SqliteLocalDb` construction — the new boot-time re-registration reads pages the old build never touched, so site-a-a's corruption (present but latent) became fatal only on upgrade. Recovery: pair stopped together, both files quarantined (`*.corrupt-2026-08-15`, kept for forensics; 394 MB / 416 MB), fresh files reseeded — site-a-b via its startup reconcile (which live-proved the row-67 fix, see row 67) and CDC snapshot resync; converged to `connected: true`, backlog 0, with the row-27 schema change proving the digest handshake green. Suspected cause is host-level, not product: see row 79. | Found live by the row-55 debug + this wave's close-out redeploy | Closed 2026-08-15. | — |
|
||||
| 70 | **→ #51** **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
|
||||
| 70 | **CLOSED 2026-11-15 → #116** (was → #51; `leader-auto-down` is now the default, so no deployed tier depends on `AutoDowning`) **`auto-down-unreachable-after` is deprecated — Akka warns "keep-majority will be used instead" on every boot** *(Medium info — verified NOT currently true, but a live removal risk)* | Row-55 live session; investigated + drill-verified at this wave's close-out | The warning is real (fires at Cluster start on every node, Akka.NET 1.5.62 — same version as the 2026-07-21 decision, so nothing changed underneath; it fires on the presence of the `auto-down-unreachable-after` key). **Drill-proven that the substitution does NOT happen when `downing-provider-class = "Akka.Cluster.AutoDowning"` is EXPLICIT (as `BuildHocon` sets):** `DRILL_MODE=active` kills central-a (oldest AND lowest address) — under keep-majority a 2-node even split keeps the lowest-address side, so surviving central-b would have downed ITSELF (total outage); instead central-b observed OldestChanged → became Oldest, took over all singletons, and the victim rejoined ready 2s after restart (2026-08-15 drill, post-merge rig). The 2026-07-21 decision therefore still holds exactly as documented. The residual risk is the deprecation itself: a future Akka.NET bump can delete `AutoDowning`, silently or loudly breaking the availability-first strategy. | **Any Akka.NET version bump** — re-run `DRILL_MODE=active bash docker/failover-drill.sh` and re-read the release notes before accepting it |
|
||||
| 71 | ~~**Standby nodes log the active-node health check at ERR every 10s** *(Low, log noise)*~~ **RESOLVED 2026-08-16 (`8b761841`).** One narrowly-keyed Serilog `Filter.ByExcluding` in `LoggerConfigurationFactory`: `Level == Error` AND the `DefaultHealthCheckService` category AND `HealthCheckName == "active-node"` — check-NAME keying is what preserves the constraint (`localdb`/`akka-cluster`/`database`/`required-singletons` keep their Error lines, and active-node has no error path of its own on the active side, so the only suppressible event is a standby reporting it's a standby). Rejected on evidence: `MinimumLevel.Override` can only raise a floor (can't rewrite Error→Information), and reporting `Degraded` would change the HTTP status `/health/active` publishes, which is load-bearing for Traefik. Bonus reach: the CENTRAL standby was logging the same Error 2x as often (Traefik probes `/health/active` every 5 s) — covered by the same filter. Fail-before: filter disabled → `StandbyActiveNodeVerdict_IsNotLogged` fails; the 9 must-still-log constraint tests pass either way, proving narrowness. Side-finding: the observed ~10 s SITE-side cadence comes from an out-of-repo prober (nothing in this repo probes a site's `/health/active`), and CLAUDE.md's "site nodes map no `/health/*`" claim was stale — corrected. | Row-55 live session (2026-08-15) | Closed 2026-08-16. | — |
|
||||
| 72 | ~~**Reconcile refreshes a STALE config but does not restart the running InstanceActor** *(Medium-Low — same silent-divergence class as row 67, arguably wider blast radius)*~~ **RESOLVED 2026-08-16 (`e606b24a`, with row 78 — one fix, the DM instance sweep).** Design decision made and recorded (`docs/plans/2026-08-15-instance-sweep-self-heal-design.md`): **converge, don't merely warn** — the sweep re-applies the stored config through **`HandleDeploy`**, the exact path a central redeploy uses, so the compile gate, warm-then-gate ordering, redeploy-supersede, static-override/native-alarm reset and all-or-nothing-per-instance are the production ones (a stop still does not cancel an in-flight run). Rationale: the stored row IS what central successfully deployed; surface-only leaves the operator to hand-issue the very redeploy the sweep would perform. Ordering hazards checked: the sweep never acts on an instance a deploy pipeline owns; a gate-rejected restart is inert, not partial; the synthetic deploy replies to `Self`, never `NoSender` (dead letters are a health metric). "Nothing surfaces it" is now false in BOTH paths — even a failed restart yields two `deployment` site events naming both revisions. Safety property beyond the brief: **one repair attempt per (instance, revision) pair**, so the self-heal can never become a restart loop for an init-crashing actor or a config the gate can never accept. Fail-before: sweep unarmed → `Sweep_RestartsInstance_ServingAStaleRevision` fails showing rev-1's value served while the store holds rev-2. | Row-67 fix session (2026-08-15) | Closed 2026-08-16. | — |
|
||||
| 73 | ~~**`SiteReconciliationActor` is silently skipped when `IDeploymentConfigFetcher` is null** *(Low)*~~ **RESOLVED 2026-08-16 (`a2f0086f`, with row 80).** New tri-state health-report field **`StartupReconcilerActive`** (`bool?`, matching the `LocalDbReplicationConnected` nullable precedent; additive proto **field 29**, vendored C# regenerated via `docker/regen-proto.sh centralcontrol` with the csproj verified restored): null = not reported (central self-report or pre-row-73 site build), false = booted with no fetcher. Latched once at boot by the Host in both branches. The skip branch was raised **Warning → Error** (no configuration legitimately omits the fetcher on a site node) plus a best-effort `deployment`/`Error` site event so evidence outlives the boot log. Central treatment: a **"No self-heal" Warning badge**, deliberately not a failure — config still arrives via replication while the node is up; what's lost is recovery from a deploy landing while it's down. Only explicit `false` is badged (badging null would light the fleet the moment central upgrades ahead of the sites); offline sites skipped, matching the backlog badge. | Row-67 fix session | Closed 2026-08-16. | — |
|
||||
|
||||
@@ -29,7 +29,7 @@ realised across two projects:
|
||||
timings, split-brain settings) bound from `appsettings.json` via the Options pattern.
|
||||
- **`src/ZB.MOM.WW.ScadaBridge.Host`** owns the cluster **bootstrap and runtime wiring**: it
|
||||
builds the Akka.NET HOCON from `ClusterOptions`, starts the `ActorSystem`,
|
||||
configures the downing strategy (`auto-down` by default; `leader-auto-down`, the in-repo port, and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
|
||||
configures the downing strategy (`leader-auto-down`, the in-repo port, by default since 2026-11-15; Akka's `auto-down` and `keep-oldest` with `down-if-alone` are the opt-in alternatives), wires
|
||||
`CoordinatedShutdown` into the service lifecycle, and provides active-node /
|
||||
cluster-membership health checks. See `Component-Host.md` (REQ-HOST-*) for detail.
|
||||
|
||||
@@ -92,7 +92,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
- Health reporting resumes from the new active node.
|
||||
- Alarm states are re-evaluated from incoming values (alarm state is in-memory only).
|
||||
|
||||
## Downing Strategy (auto-down — availability-first)
|
||||
## Downing Strategy (availability-first: `leader-auto-down`, formerly `auto-down`)
|
||||
|
||||
**Since 2026-11-15 (ScadaBridge#116) the shipped default is `leader-auto-down`** (next section). It has the same availability-first semantics as the `auto-down` decision below, but uses the in-repo provider instead of Akka's deprecated one. Every deployed tier (DEV vd03, QA vq03/vq04) and both docker rigs set it explicitly. `auto-down` stays a supported value for comparison and rollback.
|
||||
|
||||
**Decision 2026-07-21 (owner decision, resolves the deferred keep-oldest topology/strategy question):** the clusters run the **`auto-down`** downing strategy — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15s). The leader among the *reachable* members downs the unreachable peer after the stability window:
|
||||
|
||||
@@ -100,9 +102,9 @@ Akka.NET cluster singletons run on the active node of their cluster and migrate
|
||||
- **The accepted trade: dual-active during a real network partition.** With both nodes alive but partitioned, each side downs the other and continues as a one-node cluster — both claim active until the partition heals and an operator restarts one side (the restarted node rejoins the other as a fresh incarnation). This trade was chosen deliberately: site pairs run one node per VM with no shared lease infrastructure (no Kubernetes, no SQL at sites) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||
- **Stable-after duration**: 15 seconds of sustained unreachability before downing. This prevents premature downing during startup, rolling restarts, or transient network blips.
|
||||
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106, opt-in, not yet the default)
|
||||
### `leader-auto-down` — the in-repo port (ScadaBridge#106; the default since 2026-11-15, #116)
|
||||
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. The default stays `auto-down` until the docker failover drill (`DRILL_MODE=active` and standby mode) passes on the new value; switching is an appsettings flip on both nodes of a pair together, and rollback is flipping back. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
Akka deprecated `AutoDowning` in 1.5.2 and will remove it in a future release. `SplitBrainResolverStrategy: leader-auto-down` selects `LeaderAutoDowningProvider` (`src/ZB.MOM.WW.ScadaBridge.Host/Actors/LeaderAutoDowningProvider.cs`), a line-for-line port of the 1.5.62 `AutoDowning`/`AutoDown`/`AutoDownBase` logic: identical leader-only, reachable-again and removed-member semantics, and the same `StableAfter` window. `BuildHocon` emits `downing-provider-class = "ZB.MOM.WW.ScadaBridge.Host.Actors.LeaderAutoDowningProvider, ZB.MOM.WW.ScadaBridge.Host"` and carries the window under `scadabridge.cluster.leader-auto-down.unreachable-after`, so no `auto-down-unreachable-after` key is emitted and Akka's boot-time deprecation warning is not logged. It became the default on 2026-11-15, after it passed the docker rig and docker-env2 drills and the QA pair's central and site kill and stop-and-hold drills. On QA, `[leader-auto-down] DOWNING` fired at +26–27 s on a held-down node, and a killed-and-restarted node failed over in about 10 s. Switching an existing pair is an appsettings flip on both nodes together, and rollback is flipping back to `auto-down`. Every downing decision is logged with a `[leader-auto-down]` prefix (unreachable observed and its window, skipped, cancelled on reachable/removed, kept pending on a non-leader, leadership changes, and `DOWNING` at Warning), so one node's log tells the whole story. Drill + rollout deploy gate: ScadaBridge#116. Plan: `docs/plans/2026-11-09-autodowning-replacement-plan.md`.
|
||||
|
||||
#### Shut-out restart (`leader-auto-down` only, ScadaBridge#132)
|
||||
|
||||
|
||||
@@ -52,19 +52,20 @@ public class ClusterOptions
|
||||
/// <summary>
|
||||
/// Downing strategy for unreachable members. Three supported values:
|
||||
/// <list type="bullet">
|
||||
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
|
||||
/// <item><c>leader-auto-down</c> (<b>default</b> since 2026-11-15, ScadaBridge#106/#116) — the
|
||||
/// in-repo <c>LeaderAutoDowningProvider</c> (Host): the same availability-first semantics as
|
||||
/// <c>auto-down</c> below, without Akka's deprecated <c>AutoDowning</c>, so it survives that
|
||||
/// provider's removal from a future Akka.NET and does not log the boot-time deprecation warning.
|
||||
/// Adds the #132 shut-out restart (<see cref="LeaderAutoDown"/>). Drilled on the docker rig,
|
||||
/// docker-env2 and the QA pair (central + site, kill and stop-and-hold) before becoming the
|
||||
/// default.</item>
|
||||
/// <item><c>auto-down</c> (the default from 2026-07-21 until 2026-11-15; still supported) — availability-first: each
|
||||
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
|
||||
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
|
||||
/// a true network partition produces two live one-node clusters (dual-active) until
|
||||
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
|
||||
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
|
||||
/// than a rare partition.</item>
|
||||
/// <item><c>leader-auto-down</c> (opt-in, ScadaBridge#106) — identical semantics to
|
||||
/// <c>auto-down</c>, implemented by the in-repo <c>LeaderAutoDowningProvider</c> (Host) instead
|
||||
/// of Akka's deprecated <c>AutoDowning</c>, so it survives that provider's removal from a
|
||||
/// future Akka.NET and does not log the boot-time deprecation warning. Not yet the default:
|
||||
/// the docker failover drill must pass on it first
|
||||
/// (<c>docs/plans/2026-11-09-autodowning-replacement-plan.md</c>).</item>
|
||||
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
|
||||
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
|
||||
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
|
||||
@@ -76,7 +77,7 @@ public class ClusterOptions
|
||||
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
|
||||
/// lowest-address node.
|
||||
/// </summary>
|
||||
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
||||
public string SplitBrainResolverStrategy { get; set; } = "leader-auto-down";
|
||||
|
||||
/// <summary>
|
||||
/// Time the cluster membership must remain stable before the split-brain
|
||||
|
||||
@@ -14,14 +14,16 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
{
|
||||
/// <summary>
|
||||
/// Downing strategies supported for ScadaBridge's two-node clusters.
|
||||
/// <c>auto-down</c> (default) survives a crash of either node at the accepted cost
|
||||
/// <c>leader-auto-down</c> (default since 2026-11-15, ScadaBridge#116) and <c>auto-down</c>
|
||||
/// survive a crash of either node at the accepted cost
|
||||
/// of dual-active during a real partition; <c>keep-oldest</c> is partition-safe but
|
||||
/// cannot survive a crash of the oldest node. Quorum strategies are rejected:
|
||||
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
|
||||
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
|
||||
/// fatal crash to the lowest-address node instead of the oldest.
|
||||
/// <c>leader-auto-down</c> (ScadaBridge#106) is the in-repo port of <c>auto-down</c>
|
||||
/// (same semantics, no dependency on Akka's deprecated <c>AutoDowning</c>).
|
||||
/// (same semantics, no dependency on Akka's deprecated <c>AutoDowning</c>), with the #132
|
||||
/// shut-out restart.
|
||||
/// </summary>
|
||||
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
|
||||
@@ -303,12 +303,12 @@ public class AkkaHostedService : IHostedLifecycleService
|
||||
/// whitespace cannot corrupt the document or be silently misparsed.
|
||||
///
|
||||
/// The downing block branches on <see cref="ClusterOptions.SplitBrainResolverStrategy"/>:
|
||||
/// <c>auto-down</c> (default; decision 2026-07-21) installs Akka's
|
||||
/// <c>auto-down</c> (the default 2026-07-21 to 2026-11-15) installs Akka's
|
||||
/// <c>AutoDowning</c> provider with <c>auto-down-unreachable-after</c> =
|
||||
/// <see cref="ClusterOptions.StableAfter"/> — the leader among the REACHABLE members
|
||||
/// downs the unreachable peer, so a crash of either node (oldest included) fails
|
||||
/// over to the survivor; the accepted trade is dual-active during a real network
|
||||
/// partition. <c>leader-auto-down</c> (ScadaBridge#106, opt-in) installs the in-repo
|
||||
/// partition. <c>leader-auto-down</c> (ScadaBridge#106; the default since 2026-11-15, #116) installs the in-repo
|
||||
/// <see cref="LeaderAutoDowningProvider"/> — the same semantics without Akka's deprecated
|
||||
/// provider — with its duration at
|
||||
/// <see cref="LeaderAutoDowningProvider.UnreachableAfterPath"/>.
|
||||
@@ -357,14 +357,14 @@ public class AkkaHostedService : IHostedLifecycleService
|
||||
var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
|
||||
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
|
||||
|
||||
// auto-down (default): AutoDowning provider — the leader among the reachable
|
||||
// auto-down (the default until 2026-11-15; still supported): AutoDowning provider — the leader among the reachable
|
||||
// members downs the unreachable peer after StableAfter, so a crash of EITHER
|
||||
// node fails over to the survivor (dual-active during a real partition is the
|
||||
// accepted trade — decision 2026-07-21). Anything else: the SBR provider with
|
||||
// the configured active-strategy (keep-oldest), which is partition-safe but
|
||||
// cannot survive a crash of the oldest node in a two-node cluster.
|
||||
//
|
||||
// leader-auto-down (ScadaBridge#106, opt-in): the in-repo LeaderAutoDowningProvider — a
|
||||
// leader-auto-down (ScadaBridge#106; the default since 2026-11-15, #116): the in-repo LeaderAutoDowningProvider — a
|
||||
// port of AutoDowning with identical semantics that survives AutoDowning's eventual
|
||||
// removal from Akka.NET. Its duration lives under our own key, so no
|
||||
// auto-down-unreachable-after is emitted (which also silences Akka's deprecation warning).
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@localhost:8081",
|
||||
"akka.tcp://scadabridge@localhost:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
"akka.tcp://scadabridge@localhost:8082",
|
||||
"akka.tcp://scadabridge@localhost:8085"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -11,10 +11,11 @@ public class ClusterOptionsTests
|
||||
{
|
||||
var options = new ClusterOptions();
|
||||
|
||||
// 'auto-down' is the default posture (decision 2026-07-21): a crash of either
|
||||
// node fails over to the survivor; dual-active during a real partition is the
|
||||
// accepted trade for pairs with no shared lease infrastructure.
|
||||
Assert.Equal("auto-down", options.SplitBrainResolverStrategy);
|
||||
// 'leader-auto-down' is the default since 2026-11-15 (ScadaBridge#116): the in-repo port
|
||||
// of the availability-first 'auto-down' posture (decision 2026-07-21), without Akka's
|
||||
// deprecated AutoDowning provider. A crash of either node fails over to the survivor.
|
||||
Assert.Equal("leader-auto-down", options.SplitBrainResolverStrategy);
|
||||
Assert.True(options.LeaderAutoDown.RestartWhenShutOut);
|
||||
Assert.Equal(TimeSpan.FromSeconds(15), options.StableAfter);
|
||||
Assert.Equal(TimeSpan.FromSeconds(2), options.HeartbeatInterval);
|
||||
Assert.Equal(TimeSpan.FromSeconds(10), options.FailureDetectionThreshold);
|
||||
|
||||
@@ -364,4 +364,27 @@ public class HoconBuilderTests
|
||||
Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
|
||||
Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ScadaBridge#116: an environment that sets no strategy gets <c>leader-auto-down</c> with
|
||||
/// <c>LeaderAutoDown</c> at its defaults — the in-repo provider, the StableAfter window under its
|
||||
/// own key, the shut-out restart ON, and no deprecated <c>auto-down-unreachable-after</c> key.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void BuildHocon_UnsetStrategy_DefaultsToLeaderAutoDown()
|
||||
{
|
||||
var cluster = new ClusterOptions { SeedNodes = DefaultCluster().SeedNodes };
|
||||
|
||||
var config = ConfigurationFactory.ParseString(AkkaHostedService.BuildHocon(
|
||||
DefaultNode(), cluster, new[] { "Central" },
|
||||
TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(15)));
|
||||
|
||||
Assert.Equal(LeaderAutoDowningProvider.ProviderClassName,
|
||||
config.GetString("akka.cluster.downing-provider-class"));
|
||||
Assert.Equal(TimeSpan.FromSeconds(15), config.GetTimeSpan(LeaderAutoDowningProvider.UnreachableAfterPath));
|
||||
Assert.True(config.GetBoolean(ShutOutRestartSettings.EnabledPath));
|
||||
Assert.Equal(TimeSpan.FromSeconds(60), config.GetTimeSpan(ShutOutRestartSettings.GracePath));
|
||||
Assert.False(config.HasPath("akka.cluster.auto-down-unreachable-after"));
|
||||
Assert.False(config.HasPath("akka.cluster.split-brain-resolver.active-strategy"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,7 +20,7 @@ namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
|
||||
/// members (<c>KeepOldest.OldestDecision</c>: <c>otherSide == 1 && thisSide >= 2</c>).
|
||||
/// With a two-node cluster the younger survivor therefore takes <c>DownReachable</c> — it downs
|
||||
/// ITSELF — so only a YOUNGER-node crash is survivable.</item>
|
||||
/// <item><c>auto-down</c> (production default, decision 2026-07-21) — the leader among
|
||||
/// <item><c>auto-down</c> (the production default from 2026-07-21 until 2026-11-15; leader-auto-down since, #116) — the leader among
|
||||
/// the reachable members downs the unreachable peer after the stability window, so a
|
||||
/// crash of EITHER node fails over to the survivor; the accepted trade is dual-active
|
||||
/// during a real network partition.</item>
|
||||
|
||||
@@ -42,7 +42,7 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
||||
string role = "Central", TimeSpan? stableAfter = null,
|
||||
int? portA = null, int? portB = null,
|
||||
TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null,
|
||||
string strategy = "auto-down")
|
||||
string strategy = "leader-auto-down")
|
||||
{
|
||||
var f = new TwoNodeClusterFixture();
|
||||
f.PortA = portA ?? GetFreeTcpPort();
|
||||
@@ -56,8 +56,9 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
||||
}
|
||||
|
||||
/// <summary>Starts a node from production HOCON; used by StartAsync and by restart-scenarios.
|
||||
/// <paramref name="strategy"/> defaults to the production posture (auto-down, decision
|
||||
/// 2026-07-21); pass "keep-oldest" to exercise the legacy SBR path.</summary>
|
||||
/// <paramref name="strategy"/> defaults to the production posture (leader-auto-down, the default
|
||||
/// since 2026-11-15, ScadaBridge#116); pass "auto-down" for Akka's AutoDowning or "keep-oldest"
|
||||
/// to exercise the legacy SBR path.</summary>
|
||||
/// <param name="port">The remoting port for the node.</param>
|
||||
/// <param name="role">The Akka role assigned to the node.</param>
|
||||
/// <param name="stableAfter">How long membership must be stable before downing decisions; defaults to a scaled test value.</param>
|
||||
@@ -67,7 +68,7 @@ public sealed class TwoNodeClusterFixture : IAsyncDisposable
|
||||
/// <returns>The started actor system.</returns>
|
||||
public ActorSystem StartNode(int port, string role, TimeSpan? stableAfter = null,
|
||||
TimeSpan? heartbeatInterval = null, TimeSpan? failureDetectionThreshold = null,
|
||||
string strategy = "auto-down")
|
||||
string strategy = "leader-auto-down")
|
||||
{
|
||||
var nodeOptions = new NodeOptions { Role = role, NodeHostname = "127.0.0.1", RemotingPort = port };
|
||||
var clusterOptions = new ClusterOptions
|
||||
|
||||
Reference in New Issue
Block a user