feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
This commit is contained in:
+14
-12
@@ -273,29 +273,31 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
|
||||
### Automated failover drill (`failover-drill.sh`)
|
||||
|
||||
```bash
|
||||
DRILL_MODE=standby bash docker/failover-drill.sh # default — survivable younger-node crash
|
||||
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — measures the registered outage gap
|
||||
DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched
|
||||
DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER
|
||||
```
|
||||
|
||||
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, because under the unified oldest-member semantics the *active* node IS the oldest, i.e. the one crash two-node keep-oldest cannot survive:
|
||||
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
|
||||
|
||||
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The survivable direction: SBR downs the crashed member and the active node keeps its singletons. Expected result: **no routing outage at all** (the active node is never touched, so `/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s stable-after; the 2s heartbeat interval is not additive). PASS = the survivor logs the member removal within `TIMEOUT_S` (default 90s) while routing stays up.
|
||||
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** Expected result: a **total central outage** until the victim container is restarted — this is the registered deferred keep-oldest decision (master tracker 2026-07-08): keep-oldest downs the partition *without* the oldest, so the younger survivor downs itself, and it cannot re-form a cluster alone (see the seed-node constraint below). The drill confirms the dark window, then recovery within ~2 min of restarting the victim. The mode exists to make the registered gap *observable*, not to pretend it is covered.
|
||||
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
|
||||
- **`DRILL_MODE=active` — kills the ACTIVE (oldest) central node.** The survivor must **take over while the victim is still down**: it auto-downs the dead oldest, becomes the oldest member itself, re-hosts all singletons, and its `/health/active` goes 200. PASS = survivor active within `TIMEOUT_S`, then Traefik routing to it. (Under the pre-2026-07-21 `keep-oldest` strategy this direction was a proven total outage — the younger survivor took `DownReachable` and downed itself, because Akka's `down-if-alone` only rescues a side with ≥ 2 members.)
|
||||
|
||||
The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
|
||||
Both modes finish by restarting the victim and confirming it rejoins as a ready standby. The drill exercises downing-on-hard-crash, S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
|
||||
|
||||
**Seed-node bootstrap constraint.** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone restarted `central-b` (with `central-a` still down) loops on `InitJoin` forever — it never reaches `Up`, and `/health/active` never returns 200. Operator recovery actions: **(1)** restart the dead first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge (the cold-start split-brain the identical-seed-order convention exists to prevent). The real remedy is the pending keep-oldest topology/strategy decision (deferred, owner: user).
|
||||
**Partition trade (accepted).** Auto-down is availability-first: in a *real network partition* (both nodes alive, link cut) each side downs the other and both run active — dual-active until an operator restarts one side after the partition heals. This was an explicit owner decision (2026-07-21): site pairs have no shared lease infrastructure to arbitrate, and a stalled system is a bigger risk than a rare partition. See `docs/plans/2026-07-21-auto-down-availability-decision.md`.
|
||||
|
||||
> **Observed results** (plan R2-01 T3):
|
||||
**Seed-node bootstrap constraint (still applies to boot-alone).** Only the FIRST seed in `Cluster:SeedNodes` may self-join to form a *new* cluster. Both central nodes list `scadabridge-central-a` first (`docker/central-node-a/appsettings.Central.json`, `docker/central-node-b/appsettings.Central.json`), so a lone *restarted* `central-b` (with `central-a` still down) loops on `InitJoin` forever. Under auto-down this no longer causes the active-crash outage (the survivor keeps running — it never restarts), but it still bites when a node must boot alone (cold start of only the non-first-seed VM, or the survivor crashing while its peer is still dead). Operator recovery: **(1)** restart the first-seed node (`central-a`) — preferred; or **(2)** restart the survivor with a self-first seed override (env `ScadaBridge__Cluster__SeedNodes__0=akka.tcp://scadabridge@<self-host>:8081`, `ScadaBridge__Cluster__SeedNodes__1=<peer>`). The repo deliberately does NOT ship self-first ordering per node: with *both* nodes self-first, a simultaneous cold start can let each self-join independently → two one-node clusters that never merge.
|
||||
|
||||
> **Observed results** (auto-down decision verification):
|
||||
>
|
||||
> **Run 2026-07-13** against a freshly-deployed cluster on `main` @ `99544985` (round-2 merged image; `active=central-a`). Both directions behaved exactly as the design predicts.
|
||||
> **Run 2026-07-21** against a freshly-deployed cluster with `SplitBrainResolverStrategy: auto-down` (first drill: `active=central-a`). Both directions recovered.
|
||||
>
|
||||
> | Direction (`DRILL_MODE`) | Outcome | Measured |
|
||||
> |--------------------------|---------|----------|
|
||||
> | `standby` (younger-node crash) | **PASS** — SBR downed+removed the crashed `central-b`; active `central-a` kept all 7 singletons; recovered on restart. | Member removed in **27s** (budget ~25s: 10s detection + 15s stable-after); **0** `/health/active` routing blips (active node never touched); routable **0s** after victim restart. |
|
||||
> | `active` (oldest-node crash) | **Outage as designed** — killing the oldest/active `central-a` made the younger `central-b` self-down (total central outage — the registered keep-oldest gap); recovered after restarting the victim, `central-b` then assuming Oldest and re-hosting all singletons. | Outage confirmed at **9s**; central routable again **4s** after restarting `central-a`. |
|
||||
> | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
|
||||
> | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
|
||||
>
|
||||
> Notes: the `standby` PASS shows the survivable direction is clean end-to-end (SBR `DownUnreachable` decision + per-singleton "Member removed" in the survivor log, zero routing interruption). The `active` result **empirically confirms the deferred keep-oldest topology gap** (master tracker 2026-07-08 / `docs/plans/2026-07-08-deferred-work-register.md`): a hard crash of the active/oldest central node is a total outage until that node (the first seed) is restarted — the remedy remains the pending topology/strategy decision. In-process envelope (`FailoverTimingTests`, plan R2-01 T4) independently measured full failover at **33.7s**.
|
||||
> Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
|
||||
|
||||
### Central Failover
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
+52
-39
@@ -1,33 +1,33 @@
|
||||
#!/usr/bin/env bash
|
||||
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
||||
#
|
||||
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
|
||||
# ACTIVE central node — but under the unified oldest-member semantics the
|
||||
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
|
||||
# survive (registered deferred user decision, master tracker 2026-07-08;
|
||||
# SbrFailoverTests.cs XML doc). Two modes:
|
||||
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
|
||||
# downing strategy (availability-first): the leader among the REACHABLE members
|
||||
# downs the unreachable peer after StableAfter, so a hard crash of EITHER
|
||||
# central node — the active/oldest included — fails over to the survivor. The
|
||||
# accepted trade (made explicitly by the owner) is dual-active during a real
|
||||
# network partition. Both drill directions therefore expect RECOVERY:
|
||||
#
|
||||
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
|
||||
# The survivable direction: SBR downs the crashed member, the active node
|
||||
# keeps its singletons, and Traefik routing never goes dark. PASS = the
|
||||
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
|
||||
# failure detection + 15s stable-after) while /health/active stays up.
|
||||
# The active node is untouched: expect zero /health/active routing blips
|
||||
# and member removal on the survivor within ~25s (10s failure detection +
|
||||
# 15s auto-down-unreachable-after).
|
||||
#
|
||||
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
|
||||
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
|
||||
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
|
||||
# cannot help — the alone-oldest is dead and cannot down itself), and the
|
||||
# self-downed survivor cannot re-form a cluster alone unless it is the
|
||||
# FIRST seed (both nodes list central-a first; only the first seed may
|
||||
# self-join). This mode measures the dark window and PASSes only when
|
||||
# central recovers AFTER the victim container is restarted. It exists to
|
||||
# make the registered gap observable — not to pretend it is covered.
|
||||
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
|
||||
# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
|
||||
# the singletons, and /health/active goes 200 on the survivor WHILE THE
|
||||
# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
|
||||
# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
|
||||
# was a total outage — the younger survivor downed ITSELF, verified live;
|
||||
# Akka's down-if-alone only rescues a side with >= 2 members.)
|
||||
#
|
||||
# Both modes finish by restarting the victim and confirming it rejoins as a
|
||||
# fresh incarnation (standby).
|
||||
set -euo pipefail
|
||||
|
||||
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
|
||||
TIMEOUT_S="${TIMEOUT_S:-90}"
|
||||
DRILL_MODE="${DRILL_MODE:-standby}"
|
||||
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
|
||||
|
||||
active_container() {
|
||||
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
|
||||
@@ -35,6 +35,7 @@ active_container() {
|
||||
else echo "ERROR: no active central node found" >&2; exit 1; fi
|
||||
}
|
||||
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
|
||||
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
|
||||
|
||||
case "$DRILL_MODE" in
|
||||
standby|active) ;;
|
||||
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
|
||||
else
|
||||
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
|
||||
fi
|
||||
SURVIVOR_PORT=$(port_of "$SURVIVOR")
|
||||
|
||||
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
|
||||
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
|
||||
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
|
||||
START=$(date +%s)
|
||||
|
||||
if [ "$DRILL_MODE" = standby ]; then
|
||||
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
|
||||
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
|
||||
BLIPS=0
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - START ))
|
||||
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
|
||||
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
|
||||
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
|
||||
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
|
||||
echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
|
||||
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > TIMEOUT_S )); then
|
||||
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
|
||||
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
else
|
||||
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
|
||||
DARK_STREAK=0
|
||||
echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - START ))
|
||||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
|
||||
if (( DARK_STREAK >= 10 )); then
|
||||
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
|
||||
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
|
||||
if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
|
||||
echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
|
||||
echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
|
||||
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
|
||||
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
|
||||
break
|
||||
if (( ELAPSED > TIMEOUT_S )); then
|
||||
echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
|
||||
docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "Confirming Traefik routes to the new active node..."
|
||||
TR_START=$(date +%s)
|
||||
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
|
||||
if (( $(date +%s) - TR_START > 60 )); then
|
||||
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
|
||||
fi
|
||||
|
||||
echo "Restarting ${VICTIM}..."
|
||||
docker start "${VICTIM}" > /dev/null
|
||||
RESTART_AT=$(date +%s)
|
||||
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
|
||||
echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
|
||||
VICTIM_PORT=$(port_of "$VICTIM")
|
||||
while true; do
|
||||
ELAPSED=$(( $(date +%s) - RESTART_AT ))
|
||||
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
|
||||
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
|
||||
if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
|
||||
echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
|
||||
break
|
||||
fi
|
||||
if (( ELAPSED > 120 )); then
|
||||
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
|
||||
echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
|
||||
exit 1
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
|
||||
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
|
||||
echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
|
||||
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
|
||||
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "keep-oldest",
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
Reference in New Issue
Block a user