diff --git a/docker/README.md b/docker/README.md index 17262a57..04c6e9ac 100644 --- a/docker/README.md +++ b/docker/README.md @@ -385,6 +385,20 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l ```bash DRILL_MODE=standby bash docker/failover-drill.sh # default — younger-node crash, active untouched DRILL_MODE=active bash docker/failover-drill.sh # oldest-node crash — survivor must TAKE OVER +DRILL_PAIR=site-a DRILL_MODE=active bash docker/failover-drill.sh # same drill on a site pair (site-a|site-b|site-c) +DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh # freeze instead of kill (see #132 before expecting a clean rejoin) +``` + +`DRILL_PAIR` defaults to `central`. A site's `/health/*` is on container port 8084, which is not published, so the script probes every pair from a throwaway `alpine` sidecar on `scadabridge-net`. `DRILL_ACTION` defaults to `kill`. `pause` (`docker pause`, then `docker unpause`) models a frozen process or VM. + +The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node. + +**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time: + +```bash +docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \ + up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b +# back to the default: the same command without the overlay file ``` The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over: @@ -419,6 +433,17 @@ Self-first ordering is safe, and the three interesting cases are covered by `Sel > | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. | > | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. | > +> **Run 2026-11-12, `leader-auto-down` (#116 rig gate).** Every #116 gate passed on central and site-a, within about 2 s of an `auto-down` baseline on the same build (`2d2e9f35`): +> +> - Active kill: takeover in 29 s (central) and 26 s (site-a). +> - Standby kill: downed in 29 s and 26 s, with 0 blips. +> - Boot logs: no deprecation warning. +> - Partition: dual-active as accepted, recovered by restarting one side. +> - Cold start of both pairs: one cluster each. +> - Site-a S&F drained on the survivor. +> +> `docker pause` of the active node leaves a permanent second active after unpause on **both** strategies (#132). Evidence: `docs/plans/2026-11-12-leader-auto-down-rig-drill.md`. +> > Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**. ### Central Failover diff --git a/docker/docker-compose.leader-auto-down.yml b/docker/docker-compose.leader-auto-down.yml new file mode 100644 index 00000000..d60fdf54 --- /dev/null +++ b/docker/docker-compose.leader-auto-down.yml @@ -0,0 +1,27 @@ +# Opt-in overlay: run the central pair and the site-a pair on the in-repo +# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106) +# instead of the shipped default `auto-down`. Used by the #116 rig failover drill. +# +# The shipped default is NOT changed by this file: the mounted appsettings still say +# `auto-down`; this overlay layers an environment override on top (env wins over JSON). +# +# A downing-provider change must reach both nodes of a pair at the same time - never +# roll it one node at a time. Apply (from the main checkout, both pairs together): +# +# docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \ +# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b +# +# Revert to the default by re-running the same command without the overlay file. +services: + central-a: + environment: + ScadaBridge__Cluster__SplitBrainResolverStrategy: "leader-auto-down" + central-b: + environment: + ScadaBridge__Cluster__SplitBrainResolverStrategy: "leader-auto-down" + site-a-a: + environment: + ScadaBridge__Cluster__SplitBrainResolverStrategy: "leader-auto-down" + site-a-b: + environment: + ScadaBridge__Cluster__SplitBrainResolverStrategy: "leader-auto-down" diff --git a/docker/failover-drill.sh b/docker/failover-drill.sh index 7dcc0512..3f723dbc 100755 --- a/docker/failover-drill.sh +++ b/docker/failover-drill.sh @@ -1,126 +1,206 @@ #!/usr/bin/env bash # Failover drill against the running docker cluster (bash docker/deploy.sh first). # -# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down' -# downing strategy (availability-first): the leader among the REACHABLE members -# downs the unreachable peer after StableAfter, so a hard crash of EITHER -# central node — the active/oldest included — fails over to the survivor. The -# accepted trade (made explicitly by the owner) is dual-active during a real -# network partition. Both drill directions therefore expect RECOVERY: +# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster runs an availability-first +# downing strategy — `auto-down` (default) or the in-repo `leader-auto-down` +# (ScadaBridge#106; opt-in overlay docker/docker-compose.leader-auto-down.yml) — +# under which the leader among the REACHABLE members downs the unreachable peer +# after StableAfter, so a hard crash of EITHER node of a pair — the active/oldest +# included — fails over to the survivor. The accepted trade (made explicitly by +# the owner) is dual-active during a real network partition. Both drill +# directions therefore expect RECOVERY: # -# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node. -# The active node is untouched: expect zero /health/active routing blips -# and member removal on the survivor within ~25s (10s failure detection + -# 15s auto-down-unreachable-after). +# DRILL_MODE=standby (default) — hits the STANDBY (younger) node. +# The active node is untouched: expect zero /health/active blips on it and +# member removal on the survivor within ~25s (10s failure detection + +# 15s down window). # -# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR -# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts -# the singletons, and /health/active goes 200 on the survivor WHILE THE -# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe -# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction -# was a total outage — the younger survivor downed ITSELF, verified live; -# Akka's down-if-alone only rescues a side with >= 2 members.) +# DRILL_MODE=active — hits the ACTIVE (oldest) node. THE SURVIVOR MUST TAKE +# OVER: it downs the dead oldest, becomes oldest itself, hosts the +# singletons, and /health/active goes 200 on the survivor WHILE THE VICTIM +# IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe margin. # -# Both modes finish by restarting the victim and confirming it rejoins as a -# fresh incarnation (standby). +# DRILL_PAIR=central (default) | site-a | site-b | site-c — which 2-node pair. +# Central health is on container port 5000 (also published as 9001/9002 and +# routed by Traefik on 9000); a site's /health/* is on container port 8084 and +# is NOT published, so the script probes every pair from a throwaway sidecar +# container on scadabridge-net (the app image ships no curl/wget). +# +# DRILL_ACTION=kill (default) | pause — `docker kill` (SIGKILL: the hard-crash +# path; a `docker stop` would take the graceful CoordinatedShutdown path and +# prove nothing) or `docker pause` (a frozen process: sockets stay open but +# nothing answers — the GC-pause / hung-VM shape). A paused victim is +# unpaused at the end instead of started; it finds it was downed, terminates +# its actor system and comes back through the container restart policy. +# +# Both modes finish by recovering the victim and confirming it rejoins as a +# fresh incarnation (standby), then that exactly one node of the pair is active. set -euo pipefail TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}" TIMEOUT_S="${TIMEOUT_S:-90}" +REJOIN_TIMEOUT_S="${REJOIN_TIMEOUT_S:-180}" DRILL_MODE="${DRILL_MODE:-standby}" - -active_container() { - if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a - elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b - else echo "ERROR: no active central node found" >&2; exit 1; fi -} -peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; } -port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; } +DRILL_PAIR="${DRILL_PAIR:-central}" +DRILL_ACTION="${DRILL_ACTION:-kill}" +PROBE_IMAGE="${PROBE_IMAGE:-alpine:latest}" case "$DRILL_MODE" in standby|active) ;; *) echo "ERROR: DRILL_MODE must be 'standby' or 'active' (was '$DRILL_MODE')" >&2; exit 2 ;; esac +case "$DRILL_PAIR" in + central) HEALTH_PORT=5000 ;; + site-a|site-b|site-c) HEALTH_PORT=8084 ;; + *) echo "ERROR: DRILL_PAIR must be central|site-a|site-b|site-c (was '$DRILL_PAIR')" >&2; exit 2 ;; +esac +case "$DRILL_ACTION" in + kill|pause) ;; + *) echo "ERROR: DRILL_ACTION must be 'kill' or 'pause' (was '$DRILL_ACTION')" >&2; exit 2 ;; +esac + +NODE_A="scadabridge-${DRILL_PAIR}-a" +NODE_B="scadabridge-${DRILL_PAIR}-b" + +PROBE="scadabridge-drill-probe-$$" +docker run -d --rm --name "$PROBE" --network scadabridge-net "$PROBE_IMAGE" sleep 3600 > /dev/null +VICTIM="" +cleanup() { + # Never leave the victim paused/dead if the script dies early. + if [ -n "$VICTIM" ]; then + state=$(docker inspect -f '{{.State.Status}}' "$VICTIM" 2>/dev/null || true) + [ "$state" = paused ] && docker unpause "$VICTIM" > /dev/null || true + [ "$state" = exited ] && docker start "$VICTIM" > /dev/null || true + fi + docker rm -f "$PROBE" > /dev/null 2>&1 || true +} +trap cleanup EXIT + +# probe -> exit 0 on 2xx. Bounded so a frozen target cannot hang us. +probe() { docker exec "$PROBE" wget -q -T 2 -O /dev/null "http://$1:${HEALTH_PORT}$2" 2>/dev/null; } +active_container() { + if probe "$NODE_A" /health/active; then echo "$NODE_A" + elif probe "$NODE_B" /health/active; then echo "$NODE_B" + else echo "ERROR: no active ${DRILL_PAIR} node found" >&2; exit 1; fi +} +peer_of() { [ "$1" = "$NODE_A" ] && echo "$NODE_B" || echo "$NODE_A"; } +recover_victim() { + if [ "$DRILL_ACTION" = pause ]; then docker unpause "$VICTIM" > /dev/null + else docker start "$VICTIM" > /dev/null; fi +} ACTIVE=$(active_container) if [ "$DRILL_MODE" = standby ]; then - VICTIM=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE" + VICTIM_SEL=$(peer_of "$ACTIVE"); SURVIVOR="$ACTIVE" else - VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE") + VICTIM_SEL="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE") fi -SURVIVOR_PORT=$(port_of "$SURVIVOR") -echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}" -KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) -docker kill "${VICTIM}" > /dev/null +echo "pair=${DRILL_PAIR} mode=${DRILL_MODE} action=${DRILL_ACTION} active=${ACTIVE} victim=${VICTIM_SEL} survivor=${SURVIVOR}" +HIT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ) +VICTIM="$VICTIM_SEL" +docker "$DRILL_ACTION" "${VICTIM}" > /dev/null START=$(date +%s) +echo "hit at ${HIT_AT}" +# Downing evidence on the survivor. Matches both providers: Akka's own +# "Marking unreachable node ... as [Down]" / "Leader is auto-downing" and the +# in-repo "[leader-auto-down] DOWNING" line. +DOWN_RE="leader-auto-down\] DOWNING|auto-downing|marking.*node.*down|member removed|is removed" +DOWN_S="" if [ "$DRILL_MODE" = standby ]; then - echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..." + echo "Standby ${DRILL_ACTION}: waiting for ${SURVIVOR} to DOWN+REMOVE the victim (budget ~25s)..." BLIPS=0 while true; do ELAPSED=$(( $(date +%s) - START )) - curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1)) - if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then - echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)." - echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)." + probe "$SURVIVOR" /health/active || BLIPS=$((BLIPS + 1)) + if [ "$DRILL_PAIR" = central ]; then curl -sf -o /dev/null -m 2 "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1)); fi + if docker logs --since "${HIT_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "$DOWN_RE"; then + DOWN_S=$ELAPSED + echo "PASS: survivor downed/removed the victim in ${ELAPSED}s (budget ~25s: 10s detection + 15s down window)." + echo "Active-node /health/active blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)." break fi if (( ELAPSED > TIMEOUT_S )); then - echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2 - docker start "${VICTIM}" > /dev/null + echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — downing did not act" >&2 exit 1 fi sleep 1 done else - echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..." + echo "Active ${DRILL_ACTION}: waiting for ${SURVIVOR} to take over as the active node (victim stays down; budget ~25s + hand-over)..." while true; do ELAPSED=$(( $(date +%s) - START )) - if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then + if [ -z "$DOWN_S" ] && docker logs --since "${HIT_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "$DOWN_RE"; then + DOWN_S=$ELAPSED; echo " survivor logged the Down at +${DOWN_S}s" + fi + if probe "$SURVIVOR" /health/active; then echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down" - echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)." + echo "(downed the dead oldest, assumed Oldest, re-hosted the singletons)." break fi if (( ELAPSED > TIMEOUT_S )); then - echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2 - docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true - docker start "${VICTIM}" > /dev/null + echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of hitting the oldest — takeover did not happen." >&2 + docker logs --since "${HIT_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|leader-auto-down|oldest|shutting down|terminated" | tail -20 >&2 || true exit 1 fi sleep 1 done - echo "Confirming Traefik routes to the new active node..." - TR_START=$(date +%s) - while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do - if (( $(date +%s) - TR_START > 60 )); then - echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2 - docker start "${VICTIM}" > /dev/null - exit 1 - fi - sleep 1 - done - echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill." + if [ -z "$DOWN_S" ] && docker logs --since "${HIT_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "$DOWN_RE"; then + DOWN_S="<=${ELAPSED}"; echo " survivor logged the Down at ${DOWN_S}s (seen on the same poll as the takeover)" + fi + if [ "$DRILL_PAIR" = central ]; then + echo "Confirming Traefik routes to the new active node..." + TR_START=$(date +%s) + while ! curl -sf -o /dev/null -m 2 "${TRAEFIK_URL}/health/active"; do + if (( $(date +%s) - TR_START > 60 )); then + echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2 + exit 1 + fi + sleep 1 + done + echo "Traefik routing recovered $(( $(date +%s) - START ))s after the hit." + fi fi -echo "Restarting ${VICTIM}..." -docker start "${VICTIM}" > /dev/null +echo "Recovering ${VICTIM} (${DRILL_ACTION} -> $([ "$DRILL_ACTION" = pause ] && echo unpause || echo start))..." +recover_victim RESTART_AT=$(date +%s) -echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..." -VICTIM_PORT=$(port_of "$VICTIM") +echo "Waiting for the victim to rejoin as a ready standby (${VICTIM} /health/ready)..." while true; do ELAPSED=$(( $(date +%s) - RESTART_AT )) - if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then - echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart." + # A paused-then-unpaused victim was downed while frozen: it must NOT be counted + # as recovered on its stale pre-pause readiness — require the survivor to see a + # 2-member cluster again (the victim's /health/ready alone can be a zombie). + if probe "$VICTIM" /health/ready && ! probe "$VICTIM" /health/active \ + && docker exec "$PROBE" wget -q -T 2 -O - "http://${SURVIVOR}:${HEALTH_PORT}/health/ready" 2>/dev/null | grep -q '"memberCount":2,"unreachableCount":0'; then + echo "Recovered: ${VICTIM} is ready as standby (rejoined) ${ELAPSED}s after recovery." break fi - if (( ELAPSED > 120 )); then - echo "FAIL: ${VICTIM} not ready 120s after restart" >&2 + if (( ELAPSED > REJOIN_TIMEOUT_S )); then + echo "FAIL: ${VICTIM} not a ready, rejoined standby ${REJOIN_TIMEOUT_S}s after recovery" >&2 exit 1 fi sleep 1 done +VICTIM="" # recovered; nothing for the trap to undo + +echo "Checking for a lingering dual-active (10 samples, 1s apart)..." +DUAL=0 +for _ in $(seq 1 10); do + a=0; b=0 + probe "$NODE_A" /health/active && a=1 + probe "$NODE_B" /health/active && b=1 + (( a + b != 1 )) && DUAL=$((DUAL + 1)) + sleep 1 +done +if (( DUAL > 0 )); then + echo "FAIL: ${DUAL}/10 samples did not show exactly one active node" >&2 + exit 1 +fi +echo "Exactly one active node in 10/10 samples." echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):" -docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true -echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary." +docker logs --since "${HIT_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "leader-auto-down|auto-downing|-> Oldest|downing|marking.*down|removed" | tail -20 || true +echo "SUMMARY pair=${DRILL_PAIR} mode=${DRILL_MODE} action=${DRILL_ACTION} down_logged_s=${DOWN_S:-n/a} rejoin_s=${ELAPSED}" +echo "Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary." diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill.md b/docs/plans/2026-11-12-leader-auto-down-rig-drill.md new file mode 100644 index 00000000..1fbc4775 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill.md @@ -0,0 +1,146 @@ +# leader-auto-down: docker rig failover drill (ScadaBridge#116) + +**Date:** 2026-11-12 (UTC 00:44–01:14 on 2026-11-13) +**Scope:** the local docker rig only (`docker/`, 8 nodes). env2, QA (vq03/vq04) and DEV (vd03) were not touched. +**Build:** `scadabridge:latest` rebuilt from `main` @ `2d2e9f35` with `bash docker/deploy.sh`. This build includes Telemetry.Serilog `2d2e9f35`, LocalDb 0.3.7 `12fa7aff` and OPC SDK `e36cd605`. +**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` (`LeaderAutoDowningProvider`, #106) on the central pair and the site-a pair. `StableAfter` 15 s, heartbeat 2 s, failure-detection threshold 10 s (rig values, unchanged). +**Baseline:** today's default `auto-down` on the same build, run first. +**Raw output:** [`2026-11-12-leader-auto-down-rig-drill/`](2026-11-12-leader-auto-down-rig-drill/) (one file per run). + +## Result + +**The #116 rig gate passes on both pairs.** Every gate #116 lists (1–4) passed on central and site-a. The timings match the `auto-down` baseline to within about 2 s. + +One extra scenario failed for both strategies: a node frozen with `docker pause`. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a **second active node** and stays that way until it is restarted. `auto-down` does exactly the same thing, so this is parity, not a regression. It is filed as **#132** (§5) and does not block this gate. + +| # | Gate | Central | Site-a | +|---|---|---|---| +| 1/2 | Kill the active (oldest) node: survivor takes over within ~25 s plus hand-over; victim rejoins as standby | **PASS**: Down at +28 s, survivor active at 29 s, Traefik at 30 s, rejoin 2 s | **PASS**: Down at +26 s, survivor active at 26 s, rejoin 11 s | +| 1/2 | Kill the standby: it is downed and removed, with no `/health/active` blips | **PASS**: Down at +29 s, 0 blips, rejoin 2 s | **PASS**: Down at +26 s, 0 blips, rejoin 1 s | +| 3 | Boot logs: no deprecation warning; `[leader-auto-down] started`, then `is unreachable … after 00:00:15`, then `DOWNING` | **PASS** on both nodes | **PASS** on both nodes | +| 4 | Partition held past the window: dual-active happens and both nodes report 200; restarting one side recovers | **PASS**: dual-active at +29 s, still there 60 s after heal, restart recovered in 12 s | **PASS**: dual-active at +27 s, still there 60 s after heal, restart recovered in 32 s | +| extra | Cold start of both pairs together (`docker kill` all four, then `docker start` all four) | **PASS**: one 2-member cluster after 6 s, 0 `DOWNING` | **PASS**: one 2-member cluster after 1 s, 0 `DOWNING` | +| extra | Site keeps collecting and store-and-forward drains across the failover | n/a | **PASS**: the survivor runs scripts and its S&F depth drops to 0, which includes 3 items the dead node left behind | +| extra | Freeze the active node (`docker pause` for about 30 s, then unpause) | Takeover **PASS** (29 s, Traefik 32 s). Victim rejoin **FAIL**: permanent dual-active | Takeover **PASS** (27 s). Victim rejoin **FAIL**: permanent dual-active | + +Each "rejoin" check requires all three: the victim's `/health/ready` is 200, its `/health/active` is 503, and the survivor reports `memberCount 2, unreachableCount 0`. After every rejoin, 10 one-second samples each showed exactly one active node. + +## 1. Method + +- **Config:** a new opt-in overlay, [`docker/docker-compose.leader-auto-down.yml`](../../docker/docker-compose.leader-auto-down.yml). It sets the strategy as an environment override on `central-a/b` and `site-a-a/b`. The mounted appsettings, and the shipped default, still say `auto-down`. Both pairs were switched together and switched back together with `up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b`. There was no rolling restart across the provider change. +- **Drill script:** [`docker/failover-drill.sh`](../../docker/failover-drill.sh) was generalised for this run: + - `DRILL_PAIR=central|site-a|site-b|site-c`. Site `/health/*` is on container port 8084, which is not published, so the script probes from a throwaway sidecar on `scadabridge-net`. + - `DRILL_ACTION=kill|pause`. + - A stricter rejoin check (above), followed by a 10-sample dual-active check. + - The central behaviour and the Traefik check are unchanged. +- **Partition:** iptables `DROP` rules in node B's network namespace, applied from a `NET_ADMIN` sidecar, cutting all traffic between B and A. Nothing else is cut, so both nodes keep SQL, Traefik and the probes. The hold was 50 s on central and 45 s on site-a, longer than the 25 s detection-plus-window. Script: [`partition.sh`](2026-11-12-leader-auto-down-rig-drill/partition.sh). +- **Data and S&F:** `/metrics` on 8084 for each site-a node: `scadabridge_site_script_runs_total`, `site_data_connections_{connected,total}`, `store_and_forward_activity_total{Queued,Delivered}` and `store_and_forward_queue_depth_items`. +- **Overview:** the local `zb-overview` (`:5320`) page, scraped as text. + +## 2. Baseline: `auto-down` on the same build + +| Run | Result | +|---|---| +| Boot logs, all 4 nodes | The `auto-down-unreachable-after … keep-majority will be used instead` warning is present on every node. This is the **positive control** for gate 3. | +| Central, active kill | Down at +27 s, survivor active at 27 s, Traefik at 30 s, rejoin 3 s, 10/10 single-active ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log)) | +| Site-a, active kill | `Leader is auto-downing` at +26 s, survivor active at 27 s, rejoin 1 s ([log](2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log)) | +| Central, active pause | Takeover at 29 s, Traefik at 34 s. Then **permanent dual-active**: still both 200 after 120 s. This is the same outcome as `leader-auto-down` ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log)) | + +The earlier recorded `auto-down` numbers (2026-07-21, `docker/README.md`) were 28 s for active takeover and 27 s for standby removal. + +## 3. `leader-auto-down` runs + +### Gate 3: boot logs + +After the overlay was applied, there were **0** deprecation lines on `central-a`, `central-b`, `site-a-a` and `site-a-b`. `site-b-a`, still on the default, still logged the warning, which serves as a negative control. Each switched node logged: + +``` +[00:50:59 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[] +[00:51:00 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[] +[00:50:55 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[] +``` + +The switch itself was a cold recreate of both pairs at the same moment. Each pair formed one 2-member cluster, with nobody downed. + +### Gate 1: central + +Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log)). The kill was at 00:51:14: + +``` +[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False) +[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[] +[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData +``` + +The survivor was active at 29 s and Traefik routed to it at 30 s. The victim rejoined as standby 2 s after `docker start`. + +Afterwards the survivor logged `lost leadership (leader is now central-a)`. This is expected. The Akka *leader* is the lowest Up address, while the *active* node is the oldest Up member, and central-b stayed active. + +Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log)): unreachable at +14 s, `DOWNING` at +29 s, **0** `/health/active` blips (node and Traefik polled every second), rejoin 2 s. + +A second central standby kill was run by mistake with its output discarded, between the two site-a kill runs. It exited without error, and the pair was healthy afterwards. It is not counted. + +### Gate 2: site-a + +Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log), [metrics](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics)). The kill was at 01:01:56. Unreachable at 01:02:07, `DOWNING` at 01:02:22 (+26 s), `Younger -> Oldest` at 01:02:23, rejoin 11 s. + +Data path across the failover: + +``` +01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3 <- active before the kill +01:02:46 site-a-b script_runs=17 dc_connected=1/2 sf_queued=12 sf_delivered=15 sf_depth=0 <- survivor, 50 s after the kill +01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16 +01:04:14 site-a-b script_runs=85 dc_connected=1/2 sf_queued=63 sf_delivered=65 +``` + +The survivor opened the data connection and kept running scripts. It delivered 3 more S&F items than it queued itself, which are the 3 pending items the dead node left in the replicated buffer, so the inherited queue drained to 0. + +`dc_connected=1/2` is the rig's steady state before and after: one of site-a's two configured connections never connects on this rig. It is not caused by the drill. + +Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log)): `DOWNING` at +26 s, **0** active blips, rejoin 1 s. Collection on the active node carried on throughout (script runs 21 → 85). + +### Gate 4: partition + +Central ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log)). Both sides saw the other unreachable at +12 s, and **both** logged `DOWNING` of the other at +27 s. Both were 200 from +29 s. After the cut was removed at +50 s, both were still 200 at every 10 s sample for 60 s, because each side has quarantined the other. + +During the dual-active, Overview showed `Split brain · …central-a:8081 vs …central-b:8081`, and each node reported `members 1`. Restarting `central-b` brought back one 2-member cluster with central-a active in **12 s**. + +Site-a ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log)): the same sequence, with dual-active from +27 s and Overview showing `Split brain`. Both site nodes had the data connection open and were running scripts, so the data was collected twice. That is the accepted trade. Restarting `site-a-b` recovered in **32 s**. + +This matches plan §6 and the in-process `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp`. + +### Cold start of both pairs together + +[Log](2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log). All four nodes were killed, then all four started. One 2-member cluster formed in 1 s on site-a and 6 s on central, with exactly one active node per pair and 0 `DOWNING` lines. The same result came from the `--force-recreate` switches at 00:50:42 and 01:01:04. + +## 4. Rig state afterwards + +Both pairs were recreated together on the default (no overlay) at about 01:14. The rig is on `auto-down` again, and the deprecation warning is back on all four nodes, which is expected for the default. + +All 8 nodes are Up. Each pair has one active node (200) and one standby (503): central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status.txt)). + +## 5. Finding: a frozen node comes back as a permanent second active (#132) + +Runs: [central LAD](2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log), [site-a LAD](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log), [central auto-down baseline](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log). + +After `docker unpause`, the frozen node (central-b in the first run) logged: + +``` +[00:53:22 WRN] … Scheduled sending of heartbeat was delayed. Previous heartbeat was sent [33472.866] ms ago … +[00:53:22 WRN] … [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 … +[00:53:28 ERR] … AssociationError … Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.] +[00:53:37 WRN] … [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (…central-b:8081) is leader +``` + +After that, both nodes reported `/health/active` 200 until the zombie was restarted. That took 180 s on central and 90 s on site-a, both times when the drill script gave up. + +The cause is in Akka. Akka.Remote 1.5.62 publishes `ThisActorSystemQuarantinedEvent` (`EndpointManager`), but Akka.Cluster 1.5.62's `ClusterCoreDaemon` subscribes only to the opposite `QuarantinedEvent`. Nothing tells a node that it has been removed. The `auto-down` baseline behaves identically, so `LeaderAutoDowningProvider` is a faithful port in this respect too. It was **not** changed. + +The fix (self-down on `ThisActorSystemQuarantinedEvent`) would also make both sides of a healed partition restart. That changes gate 4's contract, so it is left for an owner decision in #132. + +## 6. Next step + +The next rollout step is **`docker-env2`**, and it **needs owner approval**. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass"). + +Decide #132 before the default flips. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log new file mode 100644 index 00000000..7ff3f1d2 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log @@ -0,0 +1,32 @@ +pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b +hit at 2026-11-13T00:48:59Z +Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +27s +PASS: scadabridge-central-b took over as active in 27s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Confirming Traefik routes to the new active node... +Traefik routing recovered 30s after the hit. +Recovering scadabridge-central-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)... +Recovered: scadabridge-central-a is ready as standby (rejoined) 3s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b): +[00:49:27 INF] [Central/scadabridge-central-b] Singleton manager started singleton actor [akka://scadabridge/user/site-call-audit-singleton/site-call-audit] +[00:49:27 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-purge] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-call-audit] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [notification-outbox] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [pending-deployment-purge] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-audit-reconciliation] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-ingest] in role [null] after 30 seconds. +[00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [kpi-history-recorder] in role [null] after 30 seconds. +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/pending-deployment-purge-singleton/pending-deployment-purge] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-ingest-singleton/audit-log-ingest] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-purge-singleton/audit-log-purge] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-audit-reconciliation-singleton/site-audit-reconciliation] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/notification-outbox-singleton/notification-outbox] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/kpi-history-recorder-singleton/kpi-history-recorder] +[00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-call-audit-singleton/site-call-audit] +SUMMARY pair=central mode=active action=kill down_logged_s=27 rejoin_s=3 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log new file mode 100644 index 00000000..8de70238 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log @@ -0,0 +1,11 @@ +pair=central mode=active action=pause active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b +hit at 2026-11-13T00:58:00Z +Active pause: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +29s +PASS: scadabridge-central-b took over as active in 29s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Confirming Traefik routes to the new active node... +Traefik routing recovered 34s after the hit. +Recovering scadabridge-central-a (pause -> unpause)... +Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)... +FAIL: scadabridge-central-a not a ready, rejoined standby 120s after recovery diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log new file mode 100644 index 00000000..382cc5f0 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log @@ -0,0 +1,20 @@ +pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b +hit at 2026-11-13T00:49:50Z +Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +27s +PASS: scadabridge-site-a-b took over as active in 27s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Recovering scadabridge-site-a-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)... +Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b): +[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down] +[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData +[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData +SUMMARY pair=site-a mode=active action=kill down_logged_s=27 rejoin_s=1 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/final-status.txt b/docs/plans/2026-11-12-leader-auto-down-rig-drill/final-status.txt new file mode 100644 index 00000000..243922ba --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/final-status.txt @@ -0,0 +1,12 @@ +central-a ready=200 active=200 +central-b ready=200 active=503 +site-a-a ready=200 active=200 +site-a-b ready=200 active=503 +site-b-a ready=200 active=200 +site-b-b ready=200 active=503 +site-c-a ready=200 active=200 +site-c-b ready=200 active=503 +central-a strategy env=0 deprecation=1 lad=0 +central-b strategy env=0 deprecation=1 lad=0 +site-a-a strategy env=0 deprecation=1 lad=0 +site-a-b strategy env=0 deprecation=1 lad=0 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log new file mode 100644 index 00000000..69f6dcc8 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log @@ -0,0 +1,35 @@ +pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b +hit at 2026-11-13T00:51:14Z +Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +29s +PASS: scadabridge-central-b took over as active in 29s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Confirming Traefik routes to the new active node... +Traefik routing recovered 30s after the hit. +Recovering scadabridge-central-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)... +Recovered: scadabridge-central-a is ready as standby (rejoined) 2s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b): +[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False) +[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[] +[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader +[00:51:42 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down] +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:43 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-central-a:8081, 1434844774)"]] +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[00:51:46 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[] +SUMMARY pair=central mode=active action=kill down_logged_s=29 rejoin_s=2 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log new file mode 100644 index 00000000..9f820677 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log @@ -0,0 +1,11 @@ +pair=central mode=active action=pause active=scadabridge-central-b victim=scadabridge-central-b survivor=scadabridge-central-a +hit at 2026-11-13T00:52:50Z +Active pause: waiting for scadabridge-central-a to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +29s +PASS: scadabridge-central-a took over as active in 29s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Confirming Traefik routes to the new active node... +Traefik routing recovered 32s after the hit. +Recovering scadabridge-central-b (pause -> unpause)... +Waiting for the victim to rejoin as a ready standby (scadabridge-central-b /health/ready)... +FAIL: scadabridge-central-b not a ready, rejoined standby 180s after recovery diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log new file mode 100644 index 00000000..9086d576 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log @@ -0,0 +1,23 @@ +01:07:22 before: scadabridge-central-a active=200 scadabridge-central-b active=503 (A ip 192.168.147.6) +01:07:23 partition ON (B drops all traffic to/from A) +01:07:51 DUAL-ACTIVE reached at +29s +01:08:12 end of hold (+50s): scadabridge-central-a active=200 scadabridge-central-b active=200 +01:08:12 partition HEALED +01:08:22 +10s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +01:08:32 +20s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +01:08:43 +30s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +01:08:53 +40s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +01:09:03 +50s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +01:09:13 +60s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200 +--- scadabridge-central-a downing lines +[01:07:35 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True) +[01:07:50 WRN] [Central/scadabridge-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-a:8081) is leader +[01:07:50 INF] [Central/scadabridge-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-b:8081] as [Down] +--- scadabridge-central-b downing lines +[01:07:35 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False) +[01:07:35 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[] +[01:07:50 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader +[01:07:50 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down] +restart-one-side recovered in 12s +central-a ready=200 active=200 +central-b ready=200 active=503 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log new file mode 100644 index 00000000..9ed481a4 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log @@ -0,0 +1,25 @@ +pair=central mode=standby action=kill active=scadabridge-central-b victim=scadabridge-central-a survivor=scadabridge-central-b +hit at 2026-11-13T00:52:03Z +Standby kill: waiting for scadabridge-central-b to DOWN+REMOVE the victim (budget ~25s)... +PASS: survivor downed/removed the victim in 29s (budget ~25s: 10s detection + 15s down window). +Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched). +Recovering scadabridge-central-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)... +Recovered: scadabridge-central-a is ready as standby (rejoined) 2s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b): +[00:52:17 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False) +[00:52:17 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[] +[00:52:32 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader +[00:52:32 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:33 INF] [Central/scadabridge-central-b] Member removed [akka.tcp://scadabridge@scadabridge-central-a:8081] +[00:52:34 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[] +SUMMARY pair=central mode=standby action=kill down_logged_s=29 rejoin_s=2 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log new file mode 100644 index 00000000..4a9f5de6 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log @@ -0,0 +1,11 @@ +01:12:12 cold: docker kill + start of central-a/b and site-a-a/b together +site-a: one 2-member cluster, both ready, +1s +central: one 2-member cluster, both ready, +6s +central-a ready=200 active=200 +central-b ready=200 active=503 +site-a-a ready=200 active=200 +site-a-b ready=200 active=503 +central-a: 0 DOWNING lines, 0 deprecation lines +central-b: 0 DOWNING lines, 0 deprecation lines +site-a-a: 0 DOWNING lines, 0 deprecation lines +site-a-b: 0 DOWNING lines, 0 deprecation lines diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log new file mode 100644 index 00000000..9b28f647 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log @@ -0,0 +1,24 @@ +pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b +hit at 2026-11-13T01:01:56Z +Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)... + survivor logged the Down at +26s +PASS: scadabridge-site-a-b took over as active in 26s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Recovering scadabridge-site-a-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)... +Recovered: scadabridge-site-a-a is ready as standby (rejoined) 11s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b): +[01:02:07 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False) +[01:02:07 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[] +[01:02:22 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader +[01:02:22 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down] +[01:02:23 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@scadabridge-site-a-a:8082, 350892847)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join. +[01:02:23 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[01:02:23 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData +[01:02:23 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-site-a-a:8082, 1416696295)"]] +[01:02:23 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData +[01:02:34 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[] +SUMMARY pair=site-a mode=active action=kill down_logged_s=26 rejoin_s=11 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics new file mode 100644 index 00000000..221b2793 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics @@ -0,0 +1,6 @@ +01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3 +01:01:55 site-a-b script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2 +01:02:46 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2 +01:02:46 site-a-b script_runs=17 dc_connected=1/2 sf_queued=12 sf_delivered=15 sf_depth=0 +01:02:53 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2 +01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16 sf_depth=2 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log new file mode 100644 index 00000000..fb061434 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log @@ -0,0 +1,8 @@ +pair=site-a mode=active action=pause active=scadabridge-site-a-b victim=scadabridge-site-a-b survivor=scadabridge-site-a-a +hit at 2026-11-13T01:04:21Z +Active pause: waiting for scadabridge-site-a-a to take over as the active node (victim stays down; budget ~25s + hand-over)... +PASS: scadabridge-site-a-a took over as active in 27s with the victim still down +(downed the dead oldest, assumed Oldest, re-hosted the singletons). +Recovering scadabridge-site-a-b (pause -> unpause)... +Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-b /health/ready)... +FAIL: scadabridge-site-a-b not a ready, rejoined standby 90s after recovery diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.metrics b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.metrics new file mode 100644 index 00000000..2b4f62f6 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.metrics @@ -0,0 +1,2 @@ +01:06:21 site-a-a script_runs=73 dc_connected=1/2 sf_queued=54 sf_delivered=75 sf_depth=2 +01:06:21 site-a-b script_runs=186 dc_connected=1/2 sf_queued=138 sf_delivered=118 sf_depth=5 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log new file mode 100644 index 00000000..a2a0ee95 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log @@ -0,0 +1,21 @@ +01:09:38 before: scadabridge-site-a-a active=200 scadabridge-site-a-b active=503 (A ip 192.168.147.4) +01:09:39 partition ON (B drops all traffic to/from A) +01:10:05 DUAL-ACTIVE reached at +27s +01:10:24 end of hold (+46s): scadabridge-site-a-a active=200 scadabridge-site-a-b active=200 +01:10:24 partition HEALED +01:10:34 +10s after heal: scadabridge-site-a-a active=200 scadabridge-site-a-b active=200 +01:11:25 +60s after heal: scadabridge-site-a-a active=200 scadabridge-site-a-b active=200 +--- scadabridge-site-a-a downing lines +[01:09:50 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-a:8082 is leader: True) +[01:10:05 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) is leader +[01:10:05 INF] [Site/scadabridge-site-a-a] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] as [Down] +--- scadabridge-site-a-b downing lines +[01:09:50 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False) +[01:09:50 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[] +[01:10:05 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader +[01:10:05 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down] +01:11:25 site-a-a script_runs=313 dc_connected=1/2 sf_queued=234 sf_delivered=259 sf_depth=0 +01:11:25 site-a-b script_runs=61 dc_connected=1/2 sf_queued=45 sf_delivered=45 sf_depth=2 +restart-one-side recovered in 32s +site-a-a ready=200 active=200 +site-a-b ready=200 active=503 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log new file mode 100644 index 00000000..99680afe --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log @@ -0,0 +1,20 @@ +pair=site-a mode=standby action=kill active=scadabridge-site-a-b victim=scadabridge-site-a-a survivor=scadabridge-site-a-b +hit at 2026-11-13T01:03:34Z +Standby kill: waiting for scadabridge-site-a-b to DOWN+REMOVE the victim (budget ~25s)... +PASS: survivor downed/removed the victim in 27s (budget ~25s: 10s detection + 15s down window). +Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched). +Recovering scadabridge-site-a-a (kill -> start)... +Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)... +Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery. +Checking for a lingering dual-active (10 samples, 1s apart)... +Exactly one active node in 10/10 samples. +Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b): +[01:03:45 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False) +[01:03:45 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[] +[01:04:00 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader +[01:04:00 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down] +[01:04:01 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[01:04:01 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082] +[01:04:02 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[] +SUMMARY pair=site-a mode=standby action=kill down_logged_s=27 rejoin_s=1 +Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary. diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.metrics b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.metrics new file mode 100644 index 00000000..6e8de7b0 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.metrics @@ -0,0 +1,4 @@ +01:02:53 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=2 +01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16 sf_depth=2 +01:04:14 site-a-a script_runs=0 dc_connected=0/0 sf_queued=0 sf_delivered=0 sf_depth=1 +01:04:14 site-a-b script_runs=85 dc_connected=1/2 sf_queued=63 sf_delivered=65 sf_depth=1 diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-central-dual-active.txt b/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-central-dual-active.txt new file mode 100644 index 00000000..940ec2a3 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-central-dual-active.txt @@ -0,0 +1,12 @@ + 8 instances · 4 clusters central Leader · central-a + Split brain · akka.tcp://scadabridge@scadabridge-central-a:8081 vs akka.tcp://scadabridge@scadabridge-central-b:8081 2 members · 0 unreachable central-a + scadabridge-central-a:5000 + Leader Active Up + All 5 required cluster singletons are reachable. leader akka.tcp://scadabridge@scadabridge-central-a:8081 · members 1 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-central-a:8081 + Open CentralUI ↗ ready 200 · active 200 central-b + scadabridge-central-b:5000 + All 5 required cluster singletons are reachable. leader akka.tcp://scadabridge@scadabridge-central-b:8081 · members 1 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-central-b:8081 + Open CentralUI ↗ ready 200 · active 200 site-a Leader · site-a-a 2 members · 0 unreachable site-a-a + Leader Active Up + Site LocalDb reachable. leader akka.tcp://scadabridge@scadabridge-site-a-a:8082 · members 2 · unreachable 0 · self akka.tcp://scadabridge@scadabridge-site-a-a:8082 + no management link ready 200 · active 200 site-a-b diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-site-a-dual-active.txt b/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-site-a-dual-active.txt new file mode 100644 index 00000000..88f76c6b --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/overview-during-site-a-dual-active.txt @@ -0,0 +1 @@ + Split brain · akka.tcp://scadabridge@scadabridge-site-a-a:8082 vs akka.tcp://scadabridge@scadabridge-site-a-b:8082 2 members · 0 unreachable site-a-a diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill/partition.sh b/docs/plans/2026-11-12-leader-auto-down-rig-drill/partition.sh new file mode 100644 index 00000000..b806b023 --- /dev/null +++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill/partition.sh @@ -0,0 +1,30 @@ +#!/usr/bin/env bash +# Evidence copy of the helper used for #116 gate 4 on 2026-11-12 (not a supported rig script). +# Prerequisites it assumed: a long-lived probe container on the rig network +# docker run -d --name sb-drill-probe --network scadabridge-net sb-drill-net:local sleep infinity +# and a local image with iptables + curl: +# printf 'FROM alpine:latest\nRUN apk add --no-cache iptables curl\n' | docker build -t sb-drill-net:local - +# partition.sh : cut Akka remoting between the two nodes of a pair (iptables in node B's netns), +# hold, heal, then watch. Leaves the dual-active in place for the caller to resolve. +set -euo pipefail +PAIR=$1; HOLD=$2 +A=scadabridge-$PAIR-a; B=scadabridge-$PAIR-b +[ "$PAIR" = central ] && HP=5000 || HP=8084 +AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A) +c() { docker exec sb-drill-probe curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/active" 2>/dev/null || echo ERR; } +echo "$(date -u +%T) before: $A active=$(c $A) $B active=$(c $B) (A ip $AIP)" +CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s) +docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP" +echo "$(date -u +%T) partition ON (B drops all traffic to/from A)" +DUAL_AT="" +while (( $(date +%s) - T0 < HOLD )); do + a=$(c $A); b=$(c $B) + if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi + sleep 1 +done +echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A) $B active=$(c $B)" +docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP" +echo "$(date -u +%T) partition HEALED" +for i in $(seq 1 6); do sleep 10; echo "$(date -u +%T) +$((i*10))s after heal: $A active=$(c $A) $B active=$(c $B)"; done +echo "--- $A downing lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6 +echo "--- $B downing lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6