cafdc0ce60
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
65 lines
7.5 KiB
Markdown
65 lines
7.5 KiB
Markdown
# leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)
|
||
|
||
**Date:** 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
|
||
**Scope:** `docker-env2` only (`scadabridge-env2-central-a/b`, `scadabridge-env2-site-x-a/b`, Traefik `:9100`). Approved by the owner as the next #116 rollout step. The primary rig (`docker/`), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays `auto-down`.
|
||
**Build:** image **`scadabridge:env2-116`** (`sha256:42b177cd4dfd…`), built with `SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh` from `b495f429` = `origin/main` `a2e9d718` plus the env2 tooling commit (no code change). `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged (`sha256:4389897ee144…` before and after).
|
||
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` on both env2 pairs, with the #132 shut-out restart (default on for that strategy, `ShutOutGrace` 60 s). `StableAfter` 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
|
||
**Baseline:** `auto-down` on the same build, run first.
|
||
**Raw output:** [`2026-11-12-leader-auto-down-env2-drill/`](2026-11-12-leader-auto-down-env2-drill/) (one file per run).
|
||
|
||
## Result
|
||
|
||
**Every #116 gate passes on both env2 pairs.** Timings match the primary-rig run (drill doc `2026-11-12-leader-auto-down-rig-drill.md` §3 and §7) to within a few seconds.
|
||
|
||
| Gate | env2 central | env2 site-x |
|
||
|---|---|---|
|
||
| Baseline `auto-down`, active kill | Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | `Leader is auto-downing` +28 s, rejoin 12 s, 10/10 |
|
||
| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
|
||
| Kill the active node | **PASS.** Unreachable +13 s, `DOWNING` +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** `DOWNING` +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
|
||
| Kill the standby | **PASS.** `DOWNING` +29 s, **0** `/health/active` blips (node and Traefik), rejoin 2 s | **PASS.** `DOWNING` +28 s, **0** blips, rejoin 1 s |
|
||
| Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started …` and `shut-out restart is ON …` on both nodes; drills show `is unreachable; it will be downed if still unreachable after 00:00:15`, then `DOWNING` | **PASS**, same |
|
||
| Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged `shut out by … tiebreak not needed — this node RESTARTS`, then `exit code 75`, and rejoined as standby 3 s after unpause. 10/10 | **PASS.** Takeover 27 s. Victim (site-x-b) `… RESTARTS`, exit 75, rejoined 8 s after unpause. 10/10 |
|
||
| Split past the window, then heal (iptables cut 50 s / 45 s) | **PASS.** Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a `STAY`, central-b `RESTART`; **only central-b restarted** (+74 s after heal). One active, 2 members at **+76 s** after heal, 10/10 stable | **PASS.** Dual-active at +26 s. Healed +46 s. site-x-a `STAY`, site-x-b `RESTART`; only site-x-b restarted (+79 s). One active, 2 members at **+80 s** after heal, 10/10 |
|
||
| Kill regression (after the freeze/split runs) | **PASS, unchanged.** Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | **PASS, unchanged.** Down +26 s, rejoin 1 s, 0 shut-out lines |
|
||
|
||
Each rejoin check requires the victim at `/health/ready` 200 and `/health/active` 503, and the survivor at `memberCount 2, unreachableCount 0`; "10/10" is ten one-second samples with exactly one active node.
|
||
|
||
The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after `exit code 75`, while the rig's central host took about 30 s. It is not a behaviour difference.
|
||
|
||
## 1. Method
|
||
|
||
- **Image isolation.** Two committed, default-preserving options (`b495f429`):
|
||
- `docker/build.sh` reads `SCADABRIDGE_IMAGE` (default `scadabridge:latest`) for the tag.
|
||
- `docker-env2/docker-compose.yml` runs `${SCADABRIDGE_IMAGE:-scadabridge:latest}` for the four nodes. Unset gives the old behaviour.
|
||
- **Bind mounts.** The compose files came from the worktree, but every run used `--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2`, so appsettings and logs stayed bind-mounted from the **main checkout**, not from the throwaway worktree. env2's appsettings are identical on `7e34d7c2` (main checkout) and `a2e9d718`.
|
||
- **Strategy switch.** New opt-in overlay [`docker-env2/docker-compose.leader-auto-down.yml`](../../docker-env2/docker-compose.leader-auto-down.yml), mirroring the rig's. It is an environment override; the mounted appsettings still say `auto-down`. Both pairs were switched in one `up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b` at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
|
||
- **Drill script.** `docker/failover-drill.sh` gained `DRILL_ENV=primary|env2` (container prefix `scadabridge-env2-`, Traefik `:9100`, pair `site-x`). The primary default is unchanged.
|
||
- **Split and heal.** The rig's `partition-heal.sh` with an optional `PREFIX` for the container names ([copy](2026-11-12-leader-auto-down-env2-drill/partition-heal.sh)). It needs the local `sb-drill-net:local` image (alpine + iptables + curl).
|
||
|
||
## 2. Limits of this run
|
||
|
||
- **env2 is not seeded.** `ScadaBridgeConfig2` has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. `docker-env2/seed-sites.sh` would bind a data connection to the windev MxGateway (`10.100.0.48:5120`); it was not run, because that reaches outside env2.
|
||
- **Transport smoke skipped.** `docker-env2/README.md` points only to `2026-05-24-second-environment-verification.md`, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 `/management` calls at 04:15–04:17). No export or import was run.
|
||
|
||
## 3. Final state: env2 stays on `leader-auto-down` (soak)
|
||
|
||
Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is **left on `leader-auto-down`** as the soak environment for the rollout ([final status](2026-11-12-leader-auto-down-env2-drill/final-status.txt)):
|
||
|
||
- All four nodes run `scadabridge:env2-116` with the overlay env set, all ready, each `memberCount 2, unreachableCount 0`.
|
||
- The active nodes are **central-b** and **site-x-b**. Traefik `:9100` answers `/health/active` 200.
|
||
- After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.
|
||
|
||
**To keep the soak state** when touching env2, always pass both the image and the overlay:
|
||
|
||
```bash
|
||
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
|
||
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
|
||
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
|
||
```
|
||
|
||
A plain `bash docker-env2/deploy.sh` rebuilds and runs `scadabridge:latest` on `auto-down`, which ends the soak. **Rollback** is the same command without the overlay file, applied to both pairs together.
|
||
|
||
## 4. Next step
|
||
|
||
QA pair vq03/vq04, which **needs owner approval**. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with `failureflag 1`, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.
|