Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill.md
T

7.5 KiB
Raw Blame History

leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)

Date: 2026-11-12 (UTC 04:11–04:27 on 2026-11-13) Scope: docker-env2 only (scadabridge-env2-central-a/b, scadabridge-env2-site-x-a/b, Traefik :9100). Approved by the owner as the next #116 rollout step. The primary rig (docker/), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays auto-down. Build: image scadabridge:env2-116 (sha256:42b177cd4dfd…), built with SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh from b495f429 = origin/main a2e9d718 plus the env2 tooling commit (no code change). scadabridge:latest, which the primary rig runs, was not rebuilt or retagged (sha256:4389897ee144… before and after). Strategy under test: ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down on both env2 pairs, with the #132 shut-out restart (default on for that strategy, ShutOutGrace 60 s). StableAfter 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig). Baseline: auto-down on the same build, run first. Raw output: 2026-11-12-leader-auto-down-env2-drill/ (one file per run).

Result

Every #116 gate passes on both env2 pairs. Timings match the primary-rig run (drill doc 2026-11-12-leader-auto-down-rig-drill.md §3 and §7) to within a few seconds.

Gate env2 central env2 site-x
Baseline auto-down, active kill Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 Leader is auto-downing +28 s, rejoin 12 s, 10/10
Baseline boot logs (positive control) Deprecation warning present on both nodes Present on both nodes
Kill the active node PASS. Unreachable +13 s, DOWNING +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 PASS. DOWNING +27 s, survivor active 27 s, rejoin 1 s, 10/10
Kill the standby PASS. DOWNING +29 s, 0 /health/active blips (node and Traefik), rejoin 2 s PASS. DOWNING +28 s, 0 blips, rejoin 1 s
Boot logs PASS. 0 deprecation lines; [leader-auto-down] started … and shut-out restart is ON … on both nodes; drills show is unreachable; it will be downed if still unreachable after 00:00:15, then DOWNING PASS, same
Freeze (DRILL_MODE=active DRILL_ACTION=pause) PASS. Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged shut out by … tiebreak not needed — this node RESTARTS, then exit code 75, and rejoined as standby 3 s after unpause. 10/10 PASS. Takeover 27 s. Victim (site-x-b) … RESTARTS, exit 75, rejoined 8 s after unpause. 10/10
Split past the window, then heal (iptables cut 50 s / 45 s) PASS. Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a STAY, central-b RESTART; only central-b restarted (+74 s after heal). One active, 2 members at +76 s after heal, 10/10 stable PASS. Dual-active at +26 s. Healed +46 s. site-x-a STAY, site-x-b RESTART; only site-x-b restarted (+79 s). One active, 2 members at +80 s after heal, 10/10
Kill regression (after the freeze/split runs) PASS, unchanged. Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node PASS, unchanged. Down +26 s, rejoin 1 s, 0 shut-out lines

Each rejoin check requires the victim at /health/ready 200 and /health/active 503, and the survivor at memberCount 2, unreachableCount 0; "10/10" is ten one-second samples with exactly one active node.

The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after exit code 75, while the rig's central host took about 30 s. It is not a behaviour difference.

1. Method

  • Image isolation. Two committed, default-preserving options (b495f429):
    • docker/build.sh reads SCADABRIDGE_IMAGE (default scadabridge:latest) for the tag.
    • docker-env2/docker-compose.yml runs ${SCADABRIDGE_IMAGE:-scadabridge:latest} for the four nodes. Unset gives the old behaviour.
  • Bind mounts. The compose files came from the worktree, but every run used --project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2, so appsettings and logs stayed bind-mounted from the main checkout, not from the throwaway worktree. env2's appsettings are identical on 7e34d7c2 (main checkout) and a2e9d718.
  • Strategy switch. New opt-in overlay docker-env2/docker-compose.leader-auto-down.yml, mirroring the rig's. It is an environment override; the mounted appsettings still say auto-down. Both pairs were switched in one up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
  • Drill script. docker/failover-drill.sh gained DRILL_ENV=primary|env2 (container prefix scadabridge-env2-, Traefik :9100, pair site-x). The primary default is unchanged.
  • Split and heal. The rig's partition-heal.sh with an optional PREFIX for the container names (copy). It needs the local sb-drill-net:local image (alpine + iptables + curl).

2. Limits of this run

  • env2 is not seeded. ScadaBridgeConfig2 has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. docker-env2/seed-sites.sh would bind a data connection to the windev MxGateway (10.100.0.48:5120); it was not run, because that reaches outside env2.
  • Transport smoke skipped. docker-env2/README.md points only to 2026-05-24-second-environment-verification.md, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 /management calls at 04:15–04:17). No export or import was run.

3. Final state: env2 stays on leader-auto-down (soak)

Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is left on leader-auto-down as the soak environment for the rollout (final status):

  • All four nodes run scadabridge:env2-116 with the overlay env set, all ready, each memberCount 2, unreachableCount 0.
  • The active nodes are central-b and site-x-b. Traefik :9100 answers /health/active 200.
  • After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.

To keep the soak state when touching env2, always pass both the image and the overlay:

SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
  -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
  up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b

A plain bash docker-env2/deploy.sh rebuilds and runs scadabridge:latest on auto-down, which ends the soak. Rollback is the same command without the overlay file, applied to both pairs together.

4. Next step

QA pair vq03/vq04, which needs owner approval. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with failureflag 1, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.