docs(cluster): #132 rig evidence - freeze and healed partition converge with no operator action; kill unchanged
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -200,7 +200,7 @@ spec for each is `docs/requirements/Component-<Name>.md`, and `README.md` carrie
|
||||
- **Inbound API key store — per node by default, shared on central pairs (2026-09-02, register row 108).** The shared `ZB.MOM.WW.Auth.ApiKeys` library stores inbound keys in **SQLite per node** unless `ScadaBridge:InboundApi:ApiKeyStore:Provider=SqlServer`, which points both central nodes at the same `zbauth` schema in the configuration database (library `ZB.MOM.WW.Auth.ApiKeys.SqlServer` 0.3.0) — under `SqlServer`, create/revoke/rotate/scope changes are cluster-wide immediately; a node's existing SQLite file is imported once at boot (insert-where-absent, safe on both nodes of a pair) and renamed `*.imported`. Central pairs **MUST** run `Provider=SqlServer` — a clustered Central left on `Sqlite` boots fine but `StartupValidator` logs a Warning, because a failover otherwise 401s every inbound client until keys are re-issued on the survivor. Both docker rigs (`docker/`, `docker-env2/`) run `SqlServer`; QA (`vq03`/`vq04`) runs `SqlServer` for both inbound keys and secrets since 2026-10-22 (deployment record `2026-10-22-wonder-app-vq03-vq04-3808b49f.md`); DEV (`vd03`, single node — not production) is not switched. The boot import's `*.imported` rename needs the importer's pooled SQLite handle released first — Windows refuses to rename an open file (fixed `6ed6800b`; pre-fix Windows builds need the manual rename in the runbook). Design: `docs/plans/2026-09-02-inbound-api-key-shared-store-design.md`; runbook: `docs/operations/inbound-api-key-reissue.md` → *Central pairs*.
|
||||
|
||||
### Cluster & Failover
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix.
|
||||
- **`auto-down` downing strategy (decision 2026-07-21 — availability over partition-safety).** Akka's `AutoDowning` provider, `auto-down-unreachable-after` = 15s: the leader among the REACHABLE members downs the unreachable peer, so a hard crash of EITHER node (active/oldest included) fails over to the survivor in ~25s. Accepted trade: a real partition → dual-active until an operator restarts one side. `keep-oldest` remains a supported `SplitBrainResolverStrategy` value (partition-safe, but an oldest-crash is a total outage — Akka's `down-if-alone` only rescues a side with ≥2 members, proven live + in 1.5.62 source). Decision record: `docs/plans/2026-07-21-auto-down-availability-decision.md`. **Ignore the boot-time Akka warning claiming "keep-majority will be used instead"** — drill-verified benign (2026-08-15, decision-doc postscript): the substitution does not happen with the explicit `AutoDowning` provider-class `BuildHocon` sets. But `AutoDowning` IS deprecated, so **every Akka.NET version bump must re-run `DRILL_MODE=active bash docker/failover-drill.sh`** (register row 70). **Replacement ready, opt-in (ScadaBridge#106):** `SplitBrainResolverStrategy: leader-auto-down` loads the in-repo `LeaderAutoDowningProvider` (Host, a line-for-line port of 1.5.62 `AutoDowning`; no deprecated key, no deprecation warning). It is NOT the default — flip only after the docker failover drill passes on it (deploy gate ScadaBridge#116). Its decisions log under the `[leader-auto-down]` prefix. **It also fixes the freeze / healed-partition dual-active (ScadaBridge#132):** a node its peer has shut out (downed + quarantined) restarts itself; when both sides are shut out the higher address restarts after `ScadaBridge:Cluster:LeaderAutoDown:ShutOutGrace` (60s). Opt-out `LeaderAutoDown:RestartWhenShutOut=false`. Any unplanned ActorSystem termination now exits the Host with code 75 (Windows SCM recovery with `failureflag 1` ignores exit 0).
|
||||
- Both nodes are seed nodes. `min-nr-of-members = 1`.
|
||||
- Failure detection: 2s heartbeat, 10s threshold. Total failover ~25s (drill-measured 2026-07-21 under auto-down: active-crash TAKEOVER in 28s, standby-crash removal in 27s with 0 routing blips — `docker/failover-drill.sh`).
|
||||
- CoordinatedShutdown for graceful singleton handover.
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
|
||||
**The #116 rig gate passes on both pairs.** Every gate #116 lists (1–4) passed on central and site-a. The timings match the `auto-down` baseline to within about 2 s.
|
||||
|
||||
One extra scenario failed for both strategies: a node frozen with `docker pause`. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a **second active node** and stays that way until it is restarted. `auto-down` does exactly the same thing, so this is parity, not a regression. It is filed as **#132** (§5) and does not block this gate.
|
||||
One extra scenario failed for both strategies: a node frozen with `docker pause`. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a **second active node** and stays that way until it is restarted. `auto-down` does exactly the same thing, so this is parity, not a regression. It is filed as **#132** (§5) and does not block this gate. **Update (same day, §7): fixed by the #132 shut-out restart; the freeze and the healed partition now converge to one active node with no operator action.**
|
||||
|
||||
| # | Gate | Central | Site-a |
|
||||
|---|---|---|---|
|
||||
@@ -143,4 +143,28 @@ The fix (self-down on `ThisActorSystemQuarantinedEvent`) would also make both si
|
||||
|
||||
The next rollout step is **`docker-env2`**, and it **needs owner approval**. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass").
|
||||
|
||||
Decide #132 before the default flips.
|
||||
Decide #132 before the default flips. **Decided and done — see §7.**
|
||||
|
||||
## 7. Re-run with the #132 shut-out restart (2026-11-12, UTC 02:11–02:25 on 2026-11-13)
|
||||
|
||||
**Build:** `scadabridge:latest` built with `docker/build.sh` from a worktree at `db8d9ffe` (the #132 fix on `main` `9b0720af`; rebased onto `a633005f` as `952db02a` before the push, with no conflicts and no code change). The four pair nodes were then recreated with the **main checkout's** compose plus `docker-compose.leader-auto-down.yml`, so the bind-mounted appsettings stayed the main checkout's. Both pairs switched together. Boot logs on all four showed `[leader-auto-down] shut-out restart is ON for …` next to `[leader-auto-down] started on …`.
|
||||
|
||||
**Fix under test** (`ShutOutRestart`, ScadaBridge#132; design in `Component-ClusterInfrastructure.md` → *Shut-out restart*): a node shut out by its peer restarts itself. If both sides are shut out, the **higher address** restarts after a 60 s grace and the lower stays. The restart uses the self-down path, and the process now exits 75.
|
||||
|
||||
| Gate | Central | Site-a |
|
||||
|---|---|---|
|
||||
| **Freeze** (`DRILL_MODE=active DRILL_ACTION=pause`): survivor takes over; after unpause, exactly one active node and the victim rejoins as standby, with no operator action | **PASS.** Takeover 27 s, Traefik 31 s. After unpause the victim logged `shut out by central-b … tiebreak not needed — this node RESTARTS` within 1 s (02:15:08), exited 75 and rejoined as standby **34 s** after unpause; 30 s of that was the central host's own shutdown. 10/10 single-active. The survivor logged `split from central-a … Giving it 00:01:00`, then `rejoined (Up) within the shut-out grace; this node stays` ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-active-pause.log)) | **PASS.** Takeover 27 s. Victim `shut out … RESTARTS` at unpause and rejoined **6 s** after unpause. 10/10 single-active ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-active-pause.log)) |
|
||||
| **Network split, then heal** (iptables cut 50 s / 45 s, past the 25 s window): during the cut the dual-active still happens; after the heal exactly one node restarts and the pair ends with one active node | **PASS.** Dual-active at +27 s (expected while cut). Healed at +51 s; both sides logged `split from …` 12 s after the heal, which was the first probe through. 60 s later both logged the tiebreak: `central-a … Tiebreak says STAY`, `central-b … Tiebreak says RESTART`. **Only central-b restarted** (+74 s after heal, RestartCount 0→1, central-a unchanged). One 2-member cluster with central-a active at **+76 s after heal**, 10/10 stable ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-partition-heal.log)) | **PASS.** Dual-active at +28 s. Healed at +46 s; `split from` +17 s after; tiebreak +77 s: site-a-a STAY, site-a-b RESTART. **Only site-a-b restarted**. One active node (site-a-a), 2 members at **+111 s after heal**; 30 s of that was site-a-b's host shutdown. 10/10 stable. site-a-a later logged `rejoined (Up) within the shut-out grace; this node stays` ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-partition-heal.log)) |
|
||||
| **Kill regression** (`DRILL_MODE=active`, `docker kill`) | **PASS, unchanged.** Down +26 s, Traefik 28 s, rejoin 2 s, 10/10 single-active. **No** shut-out lines on either node: the victim came back as a new incarnation ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-active-kill.log)) | **PASS, unchanged.** Down +28 s, rejoin 1 s, no shut-out lines ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-active-kill.log)) |
|
||||
|
||||
For comparison, §3 measured a kill takeover of 29 s / 26 s, and §5 recorded a permanent dual-active after a freeze or a healed partition.
|
||||
|
||||
**What the rig confirmed about the signals.** In the freeze case the victim gets S1 (`ThisActorSystemQuarantinedEvent`, logged as `The remote system has quarantined this system`) while it still counts the survivor as Up, so it restarts at once. The survivor gets S2 (`has a UID that has been quarantined`) when its probe reaches the old incarnation, and waits. In the healed partition **neither** side gets S1; both get S2 from their probes, and the tiebreak decides. This matches the in-process `ShutOutRestartScenarioTests`.
|
||||
|
||||
**Gate 4 contract change.** The dual-active still appears *during* a partition longer than the window, which is the accepted trade. After the heal it now clears itself in about one probe interval plus the 60 s grace plus one restart. Previously it waited for an operator.
|
||||
|
||||
**Timings to expect.** Freeze: one restart of the frozen node, in seconds, plus its host's shutdown time (central's host took about 30 s to stop here; sites took less than 1 s). Healed partition: probe back-off (5 → 60 s, so up to 60 s after a long cut) + 60 s grace + one restart of the higher-address node.
|
||||
|
||||
**Rig state afterwards.** All 8 nodes were recreated on the **default `auto-down`** with no overlay, running image `db8d9ffe`. All 8 are ready. Each pair has one active node and `memberCount 2, unreachableCount 0`: central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status-132.txt)). The deprecation warning is back on the default nodes, and no `shut-out restart` line appears, because the actor exists only under `leader-auto-down`.
|
||||
|
||||
Helper for the split gate: [`partition-heal.sh`](2026-11-12-leader-auto-down-rig-drill/partition-heal.sh). It is an evidence copy, not a supported rig script.
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
central-a ready=200 active=200 "memberCount":2 "unreachableCount":0
|
||||
central-b ready=200 active=503 "memberCount":2 "unreachableCount":0
|
||||
site-a-a ready=200 active=200 "memberCount":2 "unreachableCount":0
|
||||
site-a-b ready=200 active=503 "memberCount":2 "unreachableCount":0
|
||||
site-b-a ready=200 active=200 "memberCount":2 "unreachableCount":0
|
||||
site-b-b ready=200 active=503 "memberCount":2 "unreachableCount":0
|
||||
site-c-a ready=200 active=200 "memberCount":2 "unreachableCount":0
|
||||
site-c-b ready=200 active=503 "memberCount":2 "unreachableCount":0
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# ScadaBridge#132 rig gate: partition a pair past the leader-auto-down window, heal it, and
|
||||
# watch the shut-out restart converge the pair WITHOUT operator action.
|
||||
# Usage: partition-heal.sh <pair> <hold_s> [watch_s]
|
||||
# Needs: sb-drill-net:local (alpine + iptables + curl) and a probe container on scadabridge-net.
|
||||
set -uo pipefail
|
||||
PAIR=$1; HOLD=$2; WATCH=${3:-300}
|
||||
A=scadabridge-$PAIR-a; B=scadabridge-$PAIR-b
|
||||
[ "$PAIR" = central ] && HP=5000 || HP=8084
|
||||
PROBE=sb132-probe
|
||||
docker rm -f $PROBE >/dev/null 2>&1
|
||||
docker run -d --name $PROBE --network scadabridge-net sb-drill-net:local sleep infinity >/dev/null
|
||||
AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A)
|
||||
c() { docker exec $PROBE curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/$2" 2>/dev/null || echo ERR; }
|
||||
mem() { docker exec $PROBE curl -s -m 2 "http://$1:$HP/health/ready" 2>/dev/null | grep -o '"memberCount":[0-9]*' | head -1; }
|
||||
rc() { docker inspect -f '{{.RestartCount}}/{{.State.StartedAt}}' $1; }
|
||||
RA0=$(rc $A); RB0=$(rc $B)
|
||||
echo "$(date -u +%T) before: $A active=$(c $A active) $B active=$(c $B active) restarts A=$RA0 B=$RB0 (A ip $AIP)"
|
||||
CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s)
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP"
|
||||
echo "$(date -u +%T) partition ON (B drops all traffic to/from A)"
|
||||
DUAL_AT=""
|
||||
while (( $(date +%s) - T0 < HOLD )); do
|
||||
a=$(c $A active); b=$(c $B active)
|
||||
if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A active) $B active=$(c $B active)"
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP"
|
||||
H0=$(date +%s); echo "$(date -u +%T) partition HEALED (+$(( H0-T0 ))s)"
|
||||
RESTARTED=""; CONVERGED=""; STABLE=0
|
||||
while (( $(date +%s) - H0 < WATCH )); do
|
||||
sleep 2
|
||||
ra=$(rc $A); rb=$(rc $B); a=$(c $A active); b=$(c $B active)
|
||||
if [ -z "$RESTARTED" ] && { [ "$ra" != "$RA0" ] || [ "$rb" != "$RB0" ]; }; then
|
||||
RESTARTED="+$(( $(date +%s)-H0 ))s after heal: A $([ "$ra" != "$RA0" ] && echo RESTARTED || echo stayed), B $([ "$rb" != "$RB0" ] && echo RESTARTED || echo stayed)"
|
||||
echo "$(date -u +%T) $RESTARTED"
|
||||
fi
|
||||
if [ -n "$RESTARTED" ] && [ "$(c $A ready)" = 200 ] && [ "$(c $B ready)" = 200 ] && { [ "$a$b" = 200503 ] || [ "$a$b" = 503200 ]; }; then
|
||||
STABLE=$((STABLE+1))
|
||||
if [ -z "$CONVERGED" ]; then CONVERGED="+$(( $(date +%s)-H0 ))s after heal"; echo "$(date -u +%T) one active ($A=$a $B=$b), $(mem $A) $(mem $B) — converged $CONVERGED"; fi
|
||||
(( STABLE >= 10 )) && break
|
||||
else
|
||||
[ -n "$CONVERGED" ] && { echo "$(date -u +%T) lost convergence: $A=$a $B=$b"; CONVERGED=""; STABLE=0; }
|
||||
fi
|
||||
done
|
||||
echo "$(date -u +%T) final: $A active=$(c $A active) ready=$(c $A ready) restarts=$(rc $A) | $B active=$(c $B active) ready=$(c $B ready) restarts=$(rc $B)"
|
||||
echo "result: restart=[$RESTARTED] converged=[$CONVERGED] stable-samples=$STABLE"
|
||||
echo "--- $A [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
|
||||
echo "--- $B [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
|
||||
docker rm -f $PROBE >/dev/null 2>&1
|
||||
@@ -0,0 +1,22 @@
|
||||
pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T02:23:11Z
|
||||
Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +26s
|
||||
PASS: scadabridge-central-b took over as active in 26s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 28s after the hit.
|
||||
Recovering scadabridge-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
Recovered: scadabridge-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
||||
[02:23:22 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[02:23:22 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[02:23:37 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[02:23:37 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
|
||||
[02:23:40 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
SUMMARY pair=central mode=active action=kill down_logged_s=26 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
--- shut-out lines since hit (scadabridge-central-b, scadabridge-central-a)
|
||||
@@ -0,0 +1,39 @@
|
||||
pair=central mode=active action=pause active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
|
||||
hit at 2026-11-13T02:14:36Z
|
||||
Active pause: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-central-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 31s after the hit.
|
||||
Recovering scadabridge-central-a (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
|
||||
Recovered: scadabridge-central-a is ready as standby (rejoined) 34s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
|
||||
[02:14:47 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[02:14:47 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[02:15:02 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[02:15:02 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
|
||||
[02:15:08 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
||||
[02:15:41 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
SUMMARY pair=central mode=active action=pause down_logged_s=27 rejoin_s=34
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
--- victim scadabridge-central-a
|
||||
[02:15:07 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True)
|
||||
[02:15:08 ERR] [Central/scadabridge-central-a] AssociationError [akka.tcp://scadabridge@scadabridge-central-a:8081] <- akka.tcp://scadabridge@scadabridge-central-b:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.] [ at Akka.Remote.EndpointReader.HandleDisas
|
||||
[02:15:08 WRN] [Central/scadabridge-central-a] Tried to associate with unreachable remote address [akka.tcp://scadabridge@scadabridge-central-b:8081]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possib
|
||||
[02:15:08 WRN] [Central/scadabridge-central-a] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-central-b:8081: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-central-a:8081) while this node still counts it as a member (Up), so only this side is shut out; tiebreak not needed — this node RESTARTS to rejoin as a f
|
||||
[02:15:08 ERR] [Central/scadabridge-central-a] AssociationError [akka.tcp://scadabridge@scadabridge-central-a:8081] -> akka.tcp://scadabridge@scadabridge-central-b:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.] [ at Akka.Remote.EndpointReader.HandleDisas
|
||||
[02:15:08 WRN] [Central/scadabridge-central-a] Tried to associate with unreachable remote address [akka.tcp://scadabridge@scadabridge-central-b:8081]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possib
|
||||
[02:15:09 FTL] [Central/scadabridge-central-a] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[02:15:39 WRN] [Central/scadabridge-central-a] ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code 75 so the service supervisor restarts it
|
||||
[02:15:40 INF] [Central/scadabridge-central-a] Starting ScadaBridge host as Central on scadabridge-central-a
|
||||
[02:15:41 INF] [Central/scadabridge-central-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-central-a:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[02:15:41 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
[02:15:42 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
--- survivor scadabridge-central-b
|
||||
[02:15:08 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itsel
|
||||
[02:15:41 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
|
||||
[02:16:08 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 rejoined (Up) within the shut-out grace; this node (akka.tcp://scadabridge@scadabridge-central-b:8081) stays
|
||||
@@ -0,0 +1,24 @@
|
||||
02:17:18 before: scadabridge-central-a active=503 scadabridge-central-b active=200 restarts A=1/2026-11-13T02:15:39.88725207Z B=0/2026-11-13T02:11:56.112373901Z (A ip 192.168.147.4)
|
||||
02:17:18 partition ON (B drops all traffic to/from A)
|
||||
02:17:45 DUAL-ACTIVE reached at +27s
|
||||
02:18:08 end of hold (+50s): scadabridge-central-a active=200 scadabridge-central-b active=200
|
||||
02:18:09 partition HEALED (+51s)
|
||||
02:19:23 +74s after heal: A stayed, B RESTARTED
|
||||
02:19:25 one active (scadabridge-central-a=200 scadabridge-central-b=503), "memberCount":2 "memberCount":2 — converged +76s after heal
|
||||
02:19:45 final: scadabridge-central-a active=200 ready=200 restarts=1/2026-11-13T02:15:39.88725207Z | scadabridge-central-b active=503 ready=200 restarts=1/2026-11-13T02:19:22.184999272Z
|
||||
result: restart=[+74s after heal: A stayed, B RESTARTED] converged=[+76s after heal] stable-samples=10
|
||||
--- scadabridge-central-a [leader-auto-down] + quarantine lines
|
||||
[02:17:30 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True)
|
||||
[02:17:45 WRN] [Central/scadabridge-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-a:8081) is leader
|
||||
[02:18:21 WRN] [Central/scadabridge-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a froze
|
||||
[02:19:21 WRN] [Central/scadabridge-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-b:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-central-a:8081) has the lo
|
||||
--- scadabridge-central-b [leader-auto-down] + quarantine lines
|
||||
[02:17:29 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
|
||||
[02:17:29 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
|
||||
[02:17:44 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
|
||||
[02:18:20 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a froze
|
||||
[02:19:20 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-central-b:8081) has the
|
||||
[02:19:20 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now none); pending unreachable=[]
|
||||
[02:19:21 FTL] [Central/scadabridge-central-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[02:19:23 INF] [Central/scadabridge-central-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-central-b:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[02:19:23 INF] [Central/scadabridge-central-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-b:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
@@ -0,0 +1,20 @@
|
||||
pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
||||
hit at 2026-11-13T02:23:53Z
|
||||
Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +28s
|
||||
PASS: scadabridge-site-a-b took over as active in 28s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-site-a-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
||||
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
||||
[02:24:05 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[02:24:05 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[02:24:20 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[02:24:20 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
[02:24:22 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
||||
SUMMARY pair=site-a mode=active action=kill down_logged_s=28 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
--- shut-out lines since hit (scadabridge-site-a-b, scadabridge-site-a-a)
|
||||
@@ -0,0 +1,31 @@
|
||||
pair=site-a mode=active action=pause active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
||||
hit at 2026-11-13T02:16:24Z
|
||||
Active pause: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-site-a-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-site-a-a (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
||||
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 6s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
||||
[02:16:35 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[02:16:35 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[02:16:50 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[02:16:50 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
||||
[02:16:56 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
||||
[02:16:58 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
||||
SUMMARY pair=site-a mode=active action=pause down_logged_s=27 rejoin_s=6
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
--- victim scadabridge-site-a-a
|
||||
[02:16:51 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-a:8082 is leader: True)
|
||||
[02:16:56 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-site-a-b:8082: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) while this node still counts it as a member (Up), so only this side is shut out; tiebreak not needed — this node R
|
||||
[02:16:56 FTL] [Site/scadabridge-site-a-a] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[02:16:56 WRN] [Site/scadabridge-site-a-a] ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code 75 so the service supervisor restarts it
|
||||
[02:16:57 INF] [Site/scadabridge-site-a-a] Starting ScadaBridge host as Site on scadabridge-site-a-a
|
||||
[02:16:57 INF] [Site/scadabridge-site-a-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-site-a-a:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[02:16:57 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
[02:16:58 INF] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
||||
--- survivor scadabridge-site-a-b
|
||||
[02:16:56 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself befo
|
||||
@@ -0,0 +1,24 @@
|
||||
02:19:58 before: scadabridge-site-a-a active=503 scadabridge-site-a-b active=200 restarts A=1/2026-11-13T02:16:56.78136603Z B=0/2026-11-13T02:11:56.101105213Z (A ip 192.168.147.7)
|
||||
02:19:59 partition ON (B drops all traffic to/from A)
|
||||
02:20:26 DUAL-ACTIVE reached at +28s
|
||||
02:20:43 end of hold (+45s): scadabridge-site-a-a active=200 scadabridge-site-a-b active=200
|
||||
02:20:44 partition HEALED (+46s)
|
||||
02:22:34 +110s after heal: A stayed, B RESTARTED
|
||||
02:22:35 one active (scadabridge-site-a-a=200 scadabridge-site-a-b=503), "memberCount":2 "memberCount":2 — converged +111s after heal
|
||||
02:22:55 final: scadabridge-site-a-a active=200 ready=200 restarts=1/2026-11-13T02:16:56.78136603Z | scadabridge-site-a-b active=503 ready=200 restarts=1/2026-11-13T02:22:33.72599839Z
|
||||
result: restart=[+110s after heal: A stayed, B RESTARTED] converged=[+111s after heal] stable-samples=10
|
||||
--- scadabridge-site-a-a [leader-auto-down] + quarantine lines
|
||||
[02:20:10 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-a:8082 is leader: True)
|
||||
[02:20:25 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) is leader
|
||||
[02:21:01 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node
|
||||
[02:22:01 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-b:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) has the lower ad
|
||||
--- scadabridge-site-a-b [leader-auto-down] + quarantine lines
|
||||
[02:20:11 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
||||
[02:20:11 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
||||
[02:20:26 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
||||
[02:21:02 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node
|
||||
[02:22:02 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) has the highe
|
||||
[02:22:02 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now none); pending unreachable=[]
|
||||
[02:22:03 FTL] [Site/scadabridge-site-a-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[02:22:34 INF] [Site/scadabridge-site-a-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-site-a-b:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[02:22:34 INF] [Site/scadabridge-site-a-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-b:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
Reference in New Issue
Block a user