Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill.md
T

65 lines
7.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)
**Date:** 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
**Scope:** `docker-env2` only (`scadabridge-env2-central-a/b`, `scadabridge-env2-site-x-a/b`, Traefik `:9100`). Approved by the owner as the next #116 rollout step. The primary rig (`docker/`), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays `auto-down`.
**Build:** image **`scadabridge:env2-116`** (`sha256:42b177cd4dfd…`), built with `SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh` from `b495f429` = `origin/main` `a2e9d718` plus the env2 tooling commit (no code change). `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged (`sha256:4389897ee144…` before and after).
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` on both env2 pairs, with the #132 shut-out restart (default on for that strategy, `ShutOutGrace` 60 s). `StableAfter` 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
**Baseline:** `auto-down` on the same build, run first.
**Raw output:** [`2026-11-12-leader-auto-down-env2-drill/`](2026-11-12-leader-auto-down-env2-drill/) (one file per run).
## Result
**Every #116 gate passes on both env2 pairs.** Timings match the primary-rig run (drill doc `2026-11-12-leader-auto-down-rig-drill.md` §3 and §7) to within a few seconds.
| Gate | env2 central | env2 site-x |
|---|---|---|
| Baseline `auto-down`, active kill | Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | `Leader is auto-downing` +28 s, rejoin 12 s, 10/10 |
| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
| Kill the active node | **PASS.** Unreachable +13 s, `DOWNING` +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** `DOWNING` +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
| Kill the standby | **PASS.** `DOWNING` +29 s, **0** `/health/active` blips (node and Traefik), rejoin 2 s | **PASS.** `DOWNING` +28 s, **0** blips, rejoin 1 s |
| Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started …` and `shut-out restart is ON …` on both nodes; drills show `is unreachable; it will be downed if still unreachable after 00:00:15`, then `DOWNING` | **PASS**, same |
| Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged `shut out by … tiebreak not needed — this node RESTARTS`, then `exit code 75`, and rejoined as standby 3 s after unpause. 10/10 | **PASS.** Takeover 27 s. Victim (site-x-b) `… RESTARTS`, exit 75, rejoined 8 s after unpause. 10/10 |
| Split past the window, then heal (iptables cut 50 s / 45 s) | **PASS.** Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a `STAY`, central-b `RESTART`; **only central-b restarted** (+74 s after heal). One active, 2 members at **+76 s** after heal, 10/10 stable | **PASS.** Dual-active at +26 s. Healed +46 s. site-x-a `STAY`, site-x-b `RESTART`; only site-x-b restarted (+79 s). One active, 2 members at **+80 s** after heal, 10/10 |
| Kill regression (after the freeze/split runs) | **PASS, unchanged.** Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | **PASS, unchanged.** Down +26 s, rejoin 1 s, 0 shut-out lines |
Each rejoin check requires the victim at `/health/ready` 200 and `/health/active` 503, and the survivor at `memberCount 2, unreachableCount 0`; "10/10" is ten one-second samples with exactly one active node.
The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after `exit code 75`, while the rig's central host took about 30 s. It is not a behaviour difference.
## 1. Method
- **Image isolation.** Two committed, default-preserving options (`b495f429`):
- `docker/build.sh` reads `SCADABRIDGE_IMAGE` (default `scadabridge:latest`) for the tag.
- `docker-env2/docker-compose.yml` runs `${SCADABRIDGE_IMAGE:-scadabridge:latest}` for the four nodes. Unset gives the old behaviour.
- **Bind mounts.** The compose files came from the worktree, but every run used `--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2`, so appsettings and logs stayed bind-mounted from the **main checkout**, not from the throwaway worktree. env2's appsettings are identical on `7e34d7c2` (main checkout) and `a2e9d718`.
- **Strategy switch.** New opt-in overlay [`docker-env2/docker-compose.leader-auto-down.yml`](../../docker-env2/docker-compose.leader-auto-down.yml), mirroring the rig's. It is an environment override; the mounted appsettings still say `auto-down`. Both pairs were switched in one `up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b` at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
- **Drill script.** `docker/failover-drill.sh` gained `DRILL_ENV=primary|env2` (container prefix `scadabridge-env2-`, Traefik `:9100`, pair `site-x`). The primary default is unchanged.
- **Split and heal.** The rig's `partition-heal.sh` with an optional `PREFIX` for the container names ([copy](2026-11-12-leader-auto-down-env2-drill/partition-heal.sh)). It needs the local `sb-drill-net:local` image (alpine + iptables + curl).
## 2. Limits of this run
- **env2 is not seeded.** `ScadaBridgeConfig2` has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. `docker-env2/seed-sites.sh` would bind a data connection to the windev MxGateway (`10.100.0.48:5120`); it was not run, because that reaches outside env2.
- **Transport smoke skipped.** `docker-env2/README.md` points only to `2026-05-24-second-environment-verification.md`, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 `/management` calls at 04:15–04:17). No export or import was run.
## 3. Final state: env2 stays on `leader-auto-down` (soak)
Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is **left on `leader-auto-down`** as the soak environment for the rollout ([final status](2026-11-12-leader-auto-down-env2-drill/final-status.txt)):
- All four nodes run `scadabridge:env2-116` with the overlay env set, all ready, each `memberCount 2, unreachableCount 0`.
- The active nodes are **central-b** and **site-x-b**. Traefik `:9100` answers `/health/active` 200.
- After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.
**To keep the soak state** when touching env2, always pass both the image and the overlay:
```bash
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
```
A plain `bash docker-env2/deploy.sh` rebuilds and runs `scadabridge:latest` on `auto-down`, which ends the soak. **Rollback** is the same command without the overlay file, applied to both pairs together.
## 4. Next step
QA pair vq03/vq04, which **needs owner approval**. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with `failureflag 1`, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.