docs(drill): leader-auto-down env2 rollout - all #116 gates pass on central and site-x; env2 stays on it as the soak (#116)
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,64 @@
|
||||
# leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)
|
||||
|
||||
**Date:** 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
|
||||
**Scope:** `docker-env2` only (`scadabridge-env2-central-a/b`, `scadabridge-env2-site-x-a/b`, Traefik `:9100`). Approved by the owner as the next #116 rollout step. The primary rig (`docker/`), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays `auto-down`.
|
||||
**Build:** image **`scadabridge:env2-116`** (`sha256:42b177cd4dfd…`), built with `SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh` from `b495f429` = `origin/main` `a2e9d718` plus the env2 tooling commit (no code change). `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged (`sha256:4389897ee144…` before and after).
|
||||
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` on both env2 pairs, with the #132 shut-out restart (default on for that strategy, `ShutOutGrace` 60 s). `StableAfter` 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
|
||||
**Baseline:** `auto-down` on the same build, run first.
|
||||
**Raw output:** [`2026-11-12-leader-auto-down-env2-drill/`](2026-11-12-leader-auto-down-env2-drill/) (one file per run).
|
||||
|
||||
## Result
|
||||
|
||||
**Every #116 gate passes on both env2 pairs.** Timings match the primary-rig run (drill doc `2026-11-12-leader-auto-down-rig-drill.md` §3 and §7) to within a few seconds.
|
||||
|
||||
| Gate | env2 central | env2 site-x |
|
||||
|---|---|---|
|
||||
| Baseline `auto-down`, active kill | Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | `Leader is auto-downing` +28 s, rejoin 12 s, 10/10 |
|
||||
| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
|
||||
| Kill the active node | **PASS.** Unreachable +13 s, `DOWNING` +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** `DOWNING` +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
|
||||
| Kill the standby | **PASS.** `DOWNING` +29 s, **0** `/health/active` blips (node and Traefik), rejoin 2 s | **PASS.** `DOWNING` +28 s, **0** blips, rejoin 1 s |
|
||||
| Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started …` and `shut-out restart is ON …` on both nodes; drills show `is unreachable; it will be downed if still unreachable after 00:00:15`, then `DOWNING` | **PASS**, same |
|
||||
| Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged `shut out by … tiebreak not needed — this node RESTARTS`, then `exit code 75`, and rejoined as standby 3 s after unpause. 10/10 | **PASS.** Takeover 27 s. Victim (site-x-b) `… RESTARTS`, exit 75, rejoined 8 s after unpause. 10/10 |
|
||||
| Split past the window, then heal (iptables cut 50 s / 45 s) | **PASS.** Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a `STAY`, central-b `RESTART`; **only central-b restarted** (+74 s after heal). One active, 2 members at **+76 s** after heal, 10/10 stable | **PASS.** Dual-active at +26 s. Healed +46 s. site-x-a `STAY`, site-x-b `RESTART`; only site-x-b restarted (+79 s). One active, 2 members at **+80 s** after heal, 10/10 |
|
||||
| Kill regression (after the freeze/split runs) | **PASS, unchanged.** Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | **PASS, unchanged.** Down +26 s, rejoin 1 s, 0 shut-out lines |
|
||||
|
||||
Each rejoin check requires the victim at `/health/ready` 200 and `/health/active` 503, and the survivor at `memberCount 2, unreachableCount 0`; "10/10" is ten one-second samples with exactly one active node.
|
||||
|
||||
The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after `exit code 75`, while the rig's central host took about 30 s. It is not a behaviour difference.
|
||||
|
||||
## 1. Method
|
||||
|
||||
- **Image isolation.** Two committed, default-preserving options (`b495f429`):
|
||||
- `docker/build.sh` reads `SCADABRIDGE_IMAGE` (default `scadabridge:latest`) for the tag.
|
||||
- `docker-env2/docker-compose.yml` runs `${SCADABRIDGE_IMAGE:-scadabridge:latest}` for the four nodes. Unset gives the old behaviour.
|
||||
- **Bind mounts.** The compose files came from the worktree, but every run used `--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2`, so appsettings and logs stayed bind-mounted from the **main checkout**, not from the throwaway worktree. env2's appsettings are identical on `7e34d7c2` (main checkout) and `a2e9d718`.
|
||||
- **Strategy switch.** New opt-in overlay [`docker-env2/docker-compose.leader-auto-down.yml`](../../docker-env2/docker-compose.leader-auto-down.yml), mirroring the rig's. It is an environment override; the mounted appsettings still say `auto-down`. Both pairs were switched in one `up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b` at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
|
||||
- **Drill script.** `docker/failover-drill.sh` gained `DRILL_ENV=primary|env2` (container prefix `scadabridge-env2-`, Traefik `:9100`, pair `site-x`). The primary default is unchanged.
|
||||
- **Split and heal.** The rig's `partition-heal.sh` with an optional `PREFIX` for the container names ([copy](2026-11-12-leader-auto-down-env2-drill/partition-heal.sh)). It needs the local `sb-drill-net:local` image (alpine + iptables + curl).
|
||||
|
||||
## 2. Limits of this run
|
||||
|
||||
- **env2 is not seeded.** `ScadaBridgeConfig2` has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. `docker-env2/seed-sites.sh` would bind a data connection to the windev MxGateway (`10.100.0.48:5120`); it was not run, because that reaches outside env2.
|
||||
- **Transport smoke skipped.** `docker-env2/README.md` points only to `2026-05-24-second-environment-verification.md`, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 `/management` calls at 04:15–04:17). No export or import was run.
|
||||
|
||||
## 3. Final state: env2 stays on `leader-auto-down` (soak)
|
||||
|
||||
Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is **left on `leader-auto-down`** as the soak environment for the rollout ([final status](2026-11-12-leader-auto-down-env2-drill/final-status.txt)):
|
||||
|
||||
- All four nodes run `scadabridge:env2-116` with the overlay env set, all ready, each `memberCount 2, unreachableCount 0`.
|
||||
- The active nodes are **central-b** and **site-x-b**. Traefik `:9100` answers `/health/active` 200.
|
||||
- After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.
|
||||
|
||||
**To keep the soak state** when touching env2, always pass both the image and the overlay:
|
||||
|
||||
```bash
|
||||
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
|
||||
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
|
||||
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
|
||||
```
|
||||
|
||||
A plain `bash docker-env2/deploy.sh` rebuilds and runs `scadabridge:latest` on `auto-down`, which ends the soak. **Rollback** is the same command without the overlay file, applied to both pairs together.
|
||||
|
||||
## 4. Next step
|
||||
|
||||
QA pair vq03/vq04, which **needs owner approval**. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with `failureflag 1`, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.
|
||||
@@ -0,0 +1,4 @@
|
||||
central-a deprecation=1
|
||||
central-b deprecation=1
|
||||
site-x-a deprecation=1
|
||||
site-x-b deprecation=1
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
env=env2 pair=central mode=active action=kill active=scadabridge-env2-central-a victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b
|
||||
hit at 2026-11-13T04:12:15Z
|
||||
Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-env2-central-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 29s after the hit.
|
||||
Recovering scadabridge-env2-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
|
||||
Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
|
||||
[04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=27 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
env=env2 pair=site-x mode=active action=kill active=scadabridge-env2-site-x-a victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b
|
||||
hit at 2026-11-13T04:13:06Z
|
||||
Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +28s
|
||||
PASS: scadabridge-env2-site-x-b took over as active in 28s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-env2-site-x-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
|
||||
Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 12s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
|
||||
[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
|
||||
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082, 1261810842)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
|
||||
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=28 rejoin_s=12
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,8 @@
|
||||
Fri Nov 13 04:27:28 UTC 2026
|
||||
image: sha256:42b177cd4dfd69e7a492d9dea003ba15d35b53cba11e6295025d1bd6b55484fd (scadabridge:env2-116, built from b495f429 = origin/main a2e9d718 + env2 tooling)
|
||||
scadabridge:latest untouched: sha256:4389897ee144a8c2cef98c5fede74b4a5bf5e4cd0cdc07788b23d58af4c6e5fc
|
||||
scadabridge-env2-central-a image=scadabridge:env2-116 state=running restarts=0 ready=200 active=503 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
|
||||
scadabridge-env2-central-b image=scadabridge:env2-116 state=running restarts=2 ready=200 active=200 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
|
||||
scadabridge-env2-site-x-a image=scadabridge:env2-116 state=running restarts=0 ready=200 active=503 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
|
||||
scadabridge-env2-site-x-b image=scadabridge:env2-116 state=running restarts=2 ready=200 active=200 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
|
||||
traefik :9100 /health/active 200
|
||||
@@ -0,0 +1,14 @@
|
||||
central-a active=200 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
|
||||
[04:14:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
[04:14:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-a:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher add
|
||||
[04:14:25 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
|
||||
central-b active=503 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
|
||||
[04:14:24 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-b:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher add
|
||||
[04:14:24 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-b:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
site-x-a active=200 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
|
||||
[04:14:23 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
[04:14:23 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address
|
||||
[04:14:24 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
|
||||
site-x-b active=503 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
|
||||
[04:14:23 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address
|
||||
[04:14:23 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
env=env2 pair=central mode=active action=kill active=scadabridge-env2-central-a victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b
|
||||
hit at 2026-11-13T04:25:04Z
|
||||
Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +28s
|
||||
PASS: scadabridge-env2-central-b took over as active in 28s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 30s after the hit.
|
||||
Recovering scadabridge-env2-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
|
||||
Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
|
||||
[04:25:16 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
|
||||
[04:25:16 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
|
||||
[04:25:31 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
|
||||
[04:25:31 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:25:35 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=28 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,35 @@
|
||||
env=env2 pair=central mode=active action=kill active=scadabridge-env2-central-a victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b
|
||||
hit at 2026-11-13T04:14:52Z
|
||||
Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +28s
|
||||
PASS: scadabridge-env2-central-b took over as active in 28s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 32s after the hit.
|
||||
Recovering scadabridge-env2-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
|
||||
Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
|
||||
[04:15:05 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
|
||||
[04:15:05 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
|
||||
[04:15:20 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
|
||||
[04:15:20 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
|
||||
[04:15:25 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=28 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,34 @@
|
||||
env=env2 pair=central mode=active action=pause active=scadabridge-env2-central-b victim=scadabridge-env2-central-b survivor=scadabridge-env2-central-a
|
||||
hit at 2026-11-13T04:17:52Z
|
||||
Active pause: waiting for scadabridge-env2-central-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-env2-central-a took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Confirming Traefik routes to the new active node...
|
||||
Traefik routing recovered 32s after the hit.
|
||||
Recovering scadabridge-env2-central-b (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-b /health/ready)...
|
||||
Recovered: scadabridge-env2-central-b is ready as standby (rejoined) 3s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-a):
|
||||
[04:18:03 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is leader: True)
|
||||
[04:18:18 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
|
||||
[04:18:18 INF] [Central/scadabridge-env2-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] as [Down]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:18:24 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
||||
SUMMARY env=env2 pair=central mode=active action=pause down_logged_s=27 rejoin_s=3
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,24 @@
|
||||
04:19:53 before: scadabridge-env2-central-a active=200 scadabridge-env2-central-b active=503 restarts A=0/2026-11-13T04:16:07.324214092Z B=1/2026-11-13T04:18:25.981120308Z (A ip 192.168.147.15)
|
||||
04:19:53 partition ON (B drops all traffic to/from A)
|
||||
04:20:19 DUAL-ACTIVE reached at +26s
|
||||
04:20:43 end of hold (+50s): scadabridge-env2-central-a active=200 scadabridge-env2-central-b active=200
|
||||
04:20:43 partition HEALED (+50s)
|
||||
04:21:57 +74s after heal: A stayed, B RESTARTED
|
||||
04:21:59 one active (scadabridge-env2-central-a=200 scadabridge-env2-central-b=503), "memberCount":2 "memberCount":2 — converged +76s after heal
|
||||
04:22:20 final: scadabridge-env2-central-a active=200 ready=200 restarts=0/2026-11-13T04:16:07.324214092Z | scadabridge-env2-central-b active=503 ready=200 restarts=2/2026-11-13T04:21:56.583595766Z
|
||||
result: restart=[+74s after heal: A stayed, B RESTARTED] converged=[+76s after heal] stable-samples=10
|
||||
--- scadabridge-env2-central-a [leader-auto-down] + quarantine lines
|
||||
[04:20:05 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is leader: True)
|
||||
[04:20:20 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
|
||||
[04:20:56 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the ti
|
||||
[04:21:56 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8
|
||||
--- scadabridge-env2-central-b [leader-auto-down] + quarantine lines
|
||||
[04:20:04 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
|
||||
[04:20:04 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
|
||||
[04:20:19 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
|
||||
[04:20:55 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the ti
|
||||
[04:21:55 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-a:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-env2-central-
|
||||
[04:21:55 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now none); pending unreachable=[]
|
||||
[04:21:56 FTL] [Central/scadabridge-env2-central-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[04:21:57 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-b:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[04:21:57 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-b:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
@@ -0,0 +1,25 @@
|
||||
env=env2 pair=central mode=standby action=kill active=scadabridge-env2-central-b victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b
|
||||
hit at 2026-11-13T04:15:38Z
|
||||
Standby kill: waiting for scadabridge-env2-central-b to DOWN+REMOVE the victim (budget ~25s)...
|
||||
PASS: survivor downed/removed the victim in 29s (budget ~25s: 10s detection + 15s down window).
|
||||
Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
|
||||
Recovering scadabridge-env2-central-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
|
||||
Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
|
||||
[04:15:51 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
|
||||
[04:15:51 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
|
||||
[04:16:06 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
|
||||
[04:16:06 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
||||
[04:16:09 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=central mode=standby action=kill down_logged_s=29 rejoin_s=2
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,29 @@
|
||||
== central-b
|
||||
[04:18:24 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] <- akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
|
||||
[04:18:24 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-env2-central-a:8081: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) while this node still counts it as a member (Up), so only this
|
||||
[04:18:24 WRN] [Central/scadabridge-env2-central-b] Dead letter: HandleListenerRegistered from [akka://scadabridge/deadLetters] to [akka://scadabridge/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fscadabridge%40scadabridge-env2-central-a%3A8081-59#67067550]
|
||||
[04:18:24 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] -> akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
|
||||
[04:18:25 ERR] [Central/scadabridge-env2-central-b] Health check akka-cluster with status Unhealthy completed after 0.0475ms with message 'Akka cluster member status: Down'
|
||||
[04:18:25 INF] [Central/scadabridge-env2-central-b] CoordinatedShutdown invoked due to [Reason:Akka.Actor.CoordinatedShutdown+ClusterDowningReason]. Exiting with [ExitCode:3].
|
||||
[04:18:25 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] -> akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
|
||||
[04:18:25 FTL] [Central/scadabridge-env2-central-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
== site-x-b
|
||||
[04:17:41 INF] [Site/scadabridge-env2-site-x-b] Request finished HTTP/1.1 GET http://scadabridge-env2-site-x-b:8084/health/active - 200 null application/json; charset=utf-8 0.2759ms
|
||||
[04:19:11 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
|
||||
[04:19:11 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) while this node still counts it as a member (Up), so only this side i
|
||||
[04:19:11 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
|
||||
[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] <- akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
|
||||
[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
|
||||
[04:19:12 INF] [Site/scadabridge-env2-site-x-b] CoordinatedShutdown invoked due to [Reason:Akka.Actor.CoordinatedShutdown+ClusterDowningReason]. Exiting with [ExitCode:3].
|
||||
[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
|
||||
== survivor central-a
|
||||
[04:18:03 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:808
|
||||
[04:18:18 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
|
||||
[04:18:24 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this no
|
||||
[04:19:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 rejoined (Up) within the shut-out grace; this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) stays
|
||||
== survivor site-x-a
|
||||
[04:18:51 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is l
|
||||
[04:19:06 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
|
||||
[04:19:12 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has
|
||||
central-b: RESTARTS=1 exit75=1
|
||||
site-x-b: RESTARTS=1 exit75=1
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
env=env2 pair=site-x mode=active action=kill active=scadabridge-env2-site-x-a victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b
|
||||
hit at 2026-11-13T04:25:49Z
|
||||
Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +26s
|
||||
PASS: scadabridge-env2-site-x-b took over as active in 26s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-env2-site-x-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
|
||||
Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
|
||||
[04:26:00 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
|
||||
[04:26:00 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
|
||||
[04:26:15 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
|
||||
[04:26:15 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
|
||||
[04:26:16 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:26:16 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:26:16 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:26:16 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:26:17 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=26 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,23 @@
|
||||
env=env2 pair=site-x mode=active action=kill active=scadabridge-env2-site-x-a victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b
|
||||
hit at 2026-11-13T04:16:22Z
|
||||
Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-env2-site-x-b took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-env2-site-x-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
|
||||
Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
|
||||
[04:16:33 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
|
||||
[04:16:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
|
||||
[04:16:48 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
|
||||
[04:16:48 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
|
||||
[04:16:49 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:16:49 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:16:49 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:16:49 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:16:50 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=27 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,22 @@
|
||||
env=env2 pair=site-x mode=active action=pause active=scadabridge-env2-site-x-b victim=scadabridge-env2-site-x-b survivor=scadabridge-env2-site-x-a
|
||||
hit at 2026-11-13T04:18:39Z
|
||||
Active pause: waiting for scadabridge-env2-site-x-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
||||
survivor logged the Down at +27s
|
||||
PASS: scadabridge-env2-site-x-a took over as active in 27s with the victim still down
|
||||
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
||||
Recovering scadabridge-env2-site-x-b (pause -> unpause)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-b /health/ready)...
|
||||
Recovered: scadabridge-env2-site-x-b is ready as standby (rejoined) 8s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-a):
|
||||
[04:18:51 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is leader: True)
|
||||
[04:19:06 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
|
||||
[04:19:06 INF] [Site/scadabridge-env2-site-x-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] as [Down]
|
||||
[04:19:07 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
|
||||
[04:19:07 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
|
||||
[04:19:07 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:19:07 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
||||
[04:19:12 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
||||
SUMMARY env=env2 pair=site-x mode=active action=pause down_logged_s=27 rejoin_s=8
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,24 @@
|
||||
04:22:29 before: scadabridge-env2-site-x-a active=200 scadabridge-env2-site-x-b active=503 restarts A=0/2026-11-13T04:17:31.539611497Z B=1/2026-11-13T04:19:13.144396278Z (A ip 192.168.147.18)
|
||||
04:22:29 partition ON (B drops all traffic to/from A)
|
||||
04:22:55 DUAL-ACTIVE reached at +26s
|
||||
04:23:14 end of hold (+45s): scadabridge-env2-site-x-a active=200 scadabridge-env2-site-x-b active=200
|
||||
04:23:15 partition HEALED (+46s)
|
||||
04:24:34 +79s after heal: A stayed, B RESTARTED
|
||||
04:24:35 one active (scadabridge-env2-site-x-a=200 scadabridge-env2-site-x-b=503), "memberCount":2 "memberCount":2 — converged +80s after heal
|
||||
04:24:55 final: scadabridge-env2-site-x-a active=200 ready=200 restarts=0/2026-11-13T04:17:31.539611497Z | scadabridge-env2-site-x-b active=503 ready=200 restarts=2/2026-11-13T04:24:33.135347935Z
|
||||
result: restart=[+79s after heal: A stayed, B RESTARTED] converged=[+80s after heal] stable-samples=10
|
||||
--- scadabridge-env2-site-x-a [leader-auto-down] + quarantine lines
|
||||
[04:22:41 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is leader: True)
|
||||
[04:22:56 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
|
||||
[04:23:32 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak
|
||||
[04:24:32 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) h
|
||||
--- scadabridge-env2-site-x-b [leader-auto-down] + quarantine lines
|
||||
[04:22:40 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
|
||||
[04:22:40 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
|
||||
[04:22:55 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
|
||||
[04:23:31 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak
|
||||
[04:24:31 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082
|
||||
[04:24:31 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now none); pending unreachable=[]
|
||||
[04:24:32 FTL] [Site/scadabridge-env2-site-x-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
||||
[04:24:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
||||
[04:24:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
||||
@@ -0,0 +1,20 @@
|
||||
env=env2 pair=site-x mode=standby action=kill active=scadabridge-env2-site-x-b victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b
|
||||
hit at 2026-11-13T04:17:02Z
|
||||
Standby kill: waiting for scadabridge-env2-site-x-b to DOWN+REMOVE the victim (budget ~25s)...
|
||||
PASS: survivor downed/removed the victim in 28s (budget ~25s: 10s detection + 15s down window).
|
||||
Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
|
||||
Recovering scadabridge-env2-site-x-a (kill -> start)...
|
||||
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
|
||||
Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
|
||||
Checking for a lingering dual-active (10 samples, 1s apart)...
|
||||
Exactly one active node in 10/10 samples.
|
||||
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
|
||||
[04:17:15 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
|
||||
[04:17:15 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
|
||||
[04:17:30 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
|
||||
[04:17:30 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
|
||||
[04:17:31 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:17:31 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
|
||||
[04:17:32 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
|
||||
SUMMARY env=env2 pair=site-x mode=standby action=kill down_logged_s=28 rejoin_s=1
|
||||
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
||||
@@ -0,0 +1,51 @@
|
||||
#!/usr/bin/env bash
|
||||
# ScadaBridge#132 rig gate: partition a pair past the leader-auto-down window, heal it, and
|
||||
# watch the shut-out restart converge the pair WITHOUT operator action.
|
||||
# Usage: partition-heal.sh <pair> <hold_s> [watch_s]
|
||||
# Needs: sb-drill-net:local (alpine + iptables + curl) and a probe container on scadabridge-net.
|
||||
set -uo pipefail
|
||||
PAIR=$1; HOLD=$2; WATCH=${3:-300}
|
||||
A=${PREFIX:-scadabridge-}$PAIR-a; B=${PREFIX:-scadabridge-}$PAIR-b
|
||||
[ "$PAIR" = central ] && HP=5000 || HP=8084
|
||||
PROBE=sb116env2-probe
|
||||
docker rm -f $PROBE >/dev/null 2>&1
|
||||
docker run -d --name $PROBE --network scadabridge-net sb-drill-net:local sleep infinity >/dev/null
|
||||
AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A)
|
||||
c() { docker exec $PROBE curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/$2" 2>/dev/null || echo ERR; }
|
||||
mem() { docker exec $PROBE curl -s -m 2 "http://$1:$HP/health/ready" 2>/dev/null | grep -o '"memberCount":[0-9]*' | head -1; }
|
||||
rc() { docker inspect -f '{{.RestartCount}}/{{.State.StartedAt}}' $1; }
|
||||
RA0=$(rc $A); RB0=$(rc $B)
|
||||
echo "$(date -u +%T) before: $A active=$(c $A active) $B active=$(c $B active) restarts A=$RA0 B=$RB0 (A ip $AIP)"
|
||||
CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s)
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP"
|
||||
echo "$(date -u +%T) partition ON (B drops all traffic to/from A)"
|
||||
DUAL_AT=""
|
||||
while (( $(date +%s) - T0 < HOLD )); do
|
||||
a=$(c $A active); b=$(c $B active)
|
||||
if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi
|
||||
sleep 1
|
||||
done
|
||||
echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A active) $B active=$(c $B active)"
|
||||
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP"
|
||||
H0=$(date +%s); echo "$(date -u +%T) partition HEALED (+$(( H0-T0 ))s)"
|
||||
RESTARTED=""; CONVERGED=""; STABLE=0
|
||||
while (( $(date +%s) - H0 < WATCH )); do
|
||||
sleep 2
|
||||
ra=$(rc $A); rb=$(rc $B); a=$(c $A active); b=$(c $B active)
|
||||
if [ -z "$RESTARTED" ] && { [ "$ra" != "$RA0" ] || [ "$rb" != "$RB0" ]; }; then
|
||||
RESTARTED="+$(( $(date +%s)-H0 ))s after heal: A $([ "$ra" != "$RA0" ] && echo RESTARTED || echo stayed), B $([ "$rb" != "$RB0" ] && echo RESTARTED || echo stayed)"
|
||||
echo "$(date -u +%T) $RESTARTED"
|
||||
fi
|
||||
if [ -n "$RESTARTED" ] && [ "$(c $A ready)" = 200 ] && [ "$(c $B ready)" = 200 ] && { [ "$a$b" = 200503 ] || [ "$a$b" = 503200 ]; }; then
|
||||
STABLE=$((STABLE+1))
|
||||
if [ -z "$CONVERGED" ]; then CONVERGED="+$(( $(date +%s)-H0 ))s after heal"; echo "$(date -u +%T) one active ($A=$a $B=$b), $(mem $A) $(mem $B) — converged $CONVERGED"; fi
|
||||
(( STABLE >= 10 )) && break
|
||||
else
|
||||
[ -n "$CONVERGED" ] && { echo "$(date -u +%T) lost convergence: $A=$a $B=$b"; CONVERGED=""; STABLE=0; }
|
||||
fi
|
||||
done
|
||||
echo "$(date -u +%T) final: $A active=$(c $A active) ready=$(c $A ready) restarts=$(rc $A) | $B active=$(c $B active) ready=$(c $B ready) restarts=$(rc $B)"
|
||||
echo "result: restart=[$RESTARTED] converged=[$CONVERGED] stable-samples=$STABLE"
|
||||
echo "--- $A [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
|
||||
echo "--- $B [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
|
||||
docker rm -f $PROBE >/dev/null 2>&1
|
||||
@@ -168,3 +168,7 @@ For comparison, §3 measured a kill takeover of 29 s / 26 s, and §5 recorded a
|
||||
**Rig state afterwards.** All 8 nodes were recreated on the **default `auto-down`** with no overlay, running image `db8d9ffe`. All 8 are ready. Each pair has one active node and `memberCount 2, unreachableCount 0`: central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status-132.txt)). The deprecation warning is back on the default nodes, and no `shut-out restart` line appears, because the actor exists only under `leader-auto-down`.
|
||||
|
||||
Helper for the split gate: [`partition-heal.sh`](2026-11-12-leader-auto-down-rig-drill/partition-heal.sh). It is an evidence copy, not a supported rig script.
|
||||
|
||||
## 8. docker-env2 (2026-11-12)
|
||||
|
||||
The owner approved the next rollout step. All gates, including the freeze and split-then-heal gates, passed on env2's central and site-x pairs, and env2 stays on `leader-auto-down` as the soak environment. Record: [`2026-11-12-leader-auto-down-env2-drill.md`](2026-11-12-leader-auto-down-env2-drill.md).
|
||||
|
||||
Reference in New Issue
Block a user