Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
7.5 KiB
leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)
Date: 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
Scope: docker-env2 only (scadabridge-env2-central-a/b, scadabridge-env2-site-x-a/b, Traefik :9100). Approved by the owner as the next #116 rollout step. The primary rig (docker/), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays auto-down.
Build: image scadabridge:env2-116 (sha256:42b177cd4dfd…), built with SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh from b495f429 = origin/main a2e9d718 plus the env2 tooling commit (no code change). scadabridge:latest, which the primary rig runs, was not rebuilt or retagged (sha256:4389897ee144… before and after).
Strategy under test: ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down on both env2 pairs, with the #132 shut-out restart (default on for that strategy, ShutOutGrace 60 s). StableAfter 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
Baseline: auto-down on the same build, run first.
Raw output: 2026-11-12-leader-auto-down-env2-drill/ (one file per run).
Result
Every #116 gate passes on both env2 pairs. Timings match the primary-rig run (drill doc 2026-11-12-leader-auto-down-rig-drill.md §3 and §7) to within a few seconds.
| Gate | env2 central | env2 site-x |
|---|---|---|
Baseline auto-down, active kill |
Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | Leader is auto-downing +28 s, rejoin 12 s, 10/10 |
| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
| Kill the active node | PASS. Unreachable +13 s, DOWNING +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 |
PASS. DOWNING +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
| Kill the standby | PASS. DOWNING +29 s, 0 /health/active blips (node and Traefik), rejoin 2 s |
PASS. DOWNING +28 s, 0 blips, rejoin 1 s |
| Boot logs | PASS. 0 deprecation lines; [leader-auto-down] started … and shut-out restart is ON … on both nodes; drills show is unreachable; it will be downed if still unreachable after 00:00:15, then DOWNING |
PASS, same |
Freeze (DRILL_MODE=active DRILL_ACTION=pause) |
PASS. Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged shut out by … tiebreak not needed — this node RESTARTS, then exit code 75, and rejoined as standby 3 s after unpause. 10/10 |
PASS. Takeover 27 s. Victim (site-x-b) … RESTARTS, exit 75, rejoined 8 s after unpause. 10/10 |
| Split past the window, then heal (iptables cut 50 s / 45 s) | PASS. Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a STAY, central-b RESTART; only central-b restarted (+74 s after heal). One active, 2 members at +76 s after heal, 10/10 stable |
PASS. Dual-active at +26 s. Healed +46 s. site-x-a STAY, site-x-b RESTART; only site-x-b restarted (+79 s). One active, 2 members at +80 s after heal, 10/10 |
| Kill regression (after the freeze/split runs) | PASS, unchanged. Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | PASS, unchanged. Down +26 s, rejoin 1 s, 0 shut-out lines |
Each rejoin check requires the victim at /health/ready 200 and /health/active 503, and the survivor at memberCount 2, unreachableCount 0; "10/10" is ten one-second samples with exactly one active node.
The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after exit code 75, while the rig's central host took about 30 s. It is not a behaviour difference.
1. Method
- Image isolation. Two committed, default-preserving options (
b495f429):docker/build.shreadsSCADABRIDGE_IMAGE(defaultscadabridge:latest) for the tag.docker-env2/docker-compose.ymlruns${SCADABRIDGE_IMAGE:-scadabridge:latest}for the four nodes. Unset gives the old behaviour.
- Bind mounts. The compose files came from the worktree, but every run used
--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2, so appsettings and logs stayed bind-mounted from the main checkout, not from the throwaway worktree. env2's appsettings are identical on7e34d7c2(main checkout) anda2e9d718. - Strategy switch. New opt-in overlay
docker-env2/docker-compose.leader-auto-down.yml, mirroring the rig's. It is an environment override; the mounted appsettings still sayauto-down. Both pairs were switched in oneup -d --force-recreate --no-deps central-a central-b site-x-a site-x-bat 04:14:11. Each pair formed one 2-member cluster with nobody downed. - Drill script.
docker/failover-drill.shgainedDRILL_ENV=primary|env2(container prefixscadabridge-env2-, Traefik:9100, pairsite-x). The primary default is unchanged. - Split and heal. The rig's
partition-heal.shwith an optionalPREFIXfor the container names (copy). It needs the localsb-drill-net:localimage (alpine + iptables + curl).
2. Limits of this run
- env2 is not seeded.
ScadaBridgeConfig2has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here.docker-env2/seed-sites.shwould bind a data connection to the windev MxGateway (10.100.0.48:5120); it was not run, because that reaches outside env2. - Transport smoke skipped.
docker-env2/README.mdpoints only to2026-05-24-second-environment-verification.md, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300/managementcalls at 04:15–04:17). No export or import was run.
3. Final state: env2 stays on leader-auto-down (soak)
Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is left on leader-auto-down as the soak environment for the rollout (final status):
- All four nodes run
scadabridge:env2-116with the overlay env set, all ready, eachmemberCount 2, unreachableCount 0. - The active nodes are central-b and site-x-b. Traefik
:9100answers/health/active200. - After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.
To keep the soak state when touching env2, always pass both the image and the overlay:
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
A plain bash docker-env2/deploy.sh rebuilds and runs scadabridge:latest on auto-down, which ends the soak. Rollback is the same command without the overlay file, applied to both pairs together.
4. Next step
QA pair vq03/vq04, which needs owner approval. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with failureflag 1, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.