ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill.md

# leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)

**Date:** 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
**Scope:** `docker-env2` only (`scadabridge-env2-central-a/b`, `scadabridge-env2-site-x-a/b`, Traefik `:9100`). Approved by the owner as the next #116 rollout step. The primary rig (`docker/`), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays `auto-down`.
**Build:** image **`scadabridge:env2-116`** (`sha256:42b177cd4dfd…`), built with `SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh` from `b495f429` = `origin/main` `a2e9d718` plus the env2 tooling commit (no code change). That commit was rebased onto `a92a418d` (docs only) as **`e8a743de`** before the push. `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged (`sha256:4389897ee144…` before and after).
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` on both env2 pairs, with the #132 shut-out restart (default on for that strategy, `ShutOutGrace` 60 s). `StableAfter` 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
**Baseline:** `auto-down` on the same build, run first.
**Raw output:** [`2026-11-12-leader-auto-down-env2-drill/`](2026-11-12-leader-auto-down-env2-drill/) (one file per run).

## Result

**Every #116 gate passes on both env2 pairs.** Timings match the primary-rig run (drill doc `2026-11-12-leader-auto-down-rig-drill.md` §3 and §7) to within a few seconds.

| Gate | env2 central | env2 site-x |
|---|---|---|
| Baseline `auto-down`, active kill | Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | `Leader is auto-downing` +28 s, rejoin 12 s, 10/10 |
| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
| Kill the active node | **PASS.** Unreachable +13 s, `DOWNING` +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** `DOWNING` +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
| Kill the standby | **PASS.** `DOWNING` +29 s, **0** `/health/active` blips (node and Traefik), rejoin 2 s | **PASS.** `DOWNING` +28 s, **0** blips, rejoin 1 s |
| Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started …` and `shut-out restart is ON …` on both nodes; drills show `is unreachable; it will be downed if still unreachable after 00:00:15`, then `DOWNING` | **PASS**, same |
| Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged `shut out by … tiebreak not needed — this node RESTARTS`, then `exit code 75`, and rejoined as standby 3 s after unpause. 10/10 | **PASS.** Takeover 27 s. Victim (site-x-b) `… RESTARTS`, exit 75, rejoined 8 s after unpause. 10/10 |
| Split past the window, then heal (iptables cut 50 s / 45 s) | **PASS.** Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a `STAY`, central-b `RESTART`; **only central-b restarted** (+74 s after heal). One active, 2 members at **+76 s** after heal, 10/10 stable | **PASS.** Dual-active at +26 s. Healed +46 s. site-x-a `STAY`, site-x-b `RESTART`; only site-x-b restarted (+79 s). One active, 2 members at **+80 s** after heal, 10/10 |
| Kill regression (after the freeze/split runs) | **PASS, unchanged.** Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | **PASS, unchanged.** Down +26 s, rejoin 1 s, 0 shut-out lines |

Each rejoin check requires the victim at `/health/ready` 200 and `/health/active` 503, and the survivor at `memberCount 2, unreachableCount 0`; "10/10" is ten one-second samples with exactly one active node.

The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after `exit code 75`, while the rig's central host took about 30 s. It is not a behaviour difference.

## 1. Method

- **Image isolation.** Two committed, default-preserving options (`e8a743de`):
  - `docker/build.sh` reads `SCADABRIDGE_IMAGE` (default `scadabridge:latest`) for the tag.
  - `docker-env2/docker-compose.yml` runs `${SCADABRIDGE_IMAGE:-scadabridge:latest}` for the four nodes. Unset gives the old behaviour.
- **Bind mounts.** The compose files came from the worktree, but every run used `--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2`, so appsettings and logs stayed bind-mounted from the **main checkout**, not from the throwaway worktree. env2's appsettings are identical on `7e34d7c2` (main checkout) and `a2e9d718`.
- **Strategy switch.** New opt-in overlay [`docker-env2/docker-compose.leader-auto-down.yml`](../../docker-env2/docker-compose.leader-auto-down.yml), mirroring the rig's. It is an environment override; the mounted appsettings still say `auto-down`. Both pairs were switched in one `up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b` at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
- **Drill script.** `docker/failover-drill.sh` gained `DRILL_ENV=primary|env2` (container prefix `scadabridge-env2-`, Traefik `:9100`, pair `site-x`). The primary default is unchanged.
- **Split and heal.** The rig's `partition-heal.sh` with an optional `PREFIX` for the container names ([copy](2026-11-12-leader-auto-down-env2-drill/partition-heal.sh)). It needs the local `sb-drill-net:local` image (alpine + iptables + curl).

## 2. Limits of this run

- **env2 is not seeded.** `ScadaBridgeConfig2` has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. `docker-env2/seed-sites.sh` would bind a data connection to the windev MxGateway (`10.100.0.48:5120`); it was not run, because that reaches outside env2. **Closed 2026-11-13: see [§5](#5-seeded-data-path-soak-2026-11-13-utc-0432-0455).**
- **Transport smoke skipped.** `docker-env2/README.md` points only to `2026-05-24-second-environment-verification.md`, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 `/management` calls at 04:15–04:17). No export or import was run.

## 3. Final state: env2 stays on `leader-auto-down` (soak)

Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is **left on `leader-auto-down`** as the soak environment for the rollout ([final status](2026-11-12-leader-auto-down-env2-drill/final-status.txt)):

- All four nodes run `scadabridge:env2-116` with the overlay env set, all ready, each `memberCount 2, unreachableCount 0`.
- The active nodes are **central-b** and **site-x-b**. Traefik `:9100` answers `/health/active` 200.
- After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.

**To keep the soak state** when touching env2, always pass both the image and the overlay:

```bash
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
  -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
  up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
```

A plain `bash docker-env2/deploy.sh` rebuilds and runs `scadabridge:latest` on `auto-down`, which ends the soak. **Rollback** is the same command without the overlay file, applied to both pairs together.

## 4. Next step

QA pair vq03/vq04, which **needs owner approval**. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with `failureflag 1`, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.

## 5. Seeded data-path soak (2026-11-13 UTC 04:32–04:55)

The gap in §2 is closed: env2 now carries real data, and the site-x failovers were re-run with it flowing. Raw output is in the same directory, prefixed `seeded-`.

### What was seeded

- `docker-env2/seed-sites.sh`, as committed: Site-X (id 3), the *Engineering Alerts* list, the MxGateway connection *ScadaBridge Site X* to the windev gateway `http://10.100.0.48:5120`, the LDAP mappings, and the `MxAlarmDemo-1` instance. The connection authenticates with the API key `mxgw_scadabridgeshared_…`, which is written into the script itself. The management API returns it as `***REDACTED***` with `hasSecrets: true`. The windev gateway was only used as a client. The LDAP step logged two harmless `2601` duplicate-name errors, because the names already exist under other ids.
- New `docker-env2/seed-data-path.sh` (commit `8b6d3268`):
  - The env2-owned sink container `scadabridge-env2-sfsink` (`docker-env2/sfsink/sink.py`). It records every POST, and answers 503 while its mode file says `fail`.
  - The OPC UA connection *OPC PLC Simulator* to the shared `scadabridge-opcua`.
  - The external system *EnvSink* (method `Record`, retry every 5 s, retry budget 100000).
  - The instance **`env-soak-1`**: `Spike` from `ns=3;s=SpikeData`, `RandomInt` from `ns=3;s=RandomSignedInt32`, and a 2 s Interval script `SoakTick`. Each run sends `ExternalSystem.CachedCall("EnvSink","Record")` with `tick = unixSeconds / 2`. The sink therefore gets one row per tick: a missing tick is either a run that never happened or a lost message, and a repeated tick is a duplicate.
  - The instance **`env-galaxy-1`**: `GalaxyInt` from `TestMachine_003.TestChangingInt`, **read only** (no write binding, no script).
- The sink's mode switch is what makes store-and-forward buffer: a 503 is transient, so the call is parked and retried. A healthy target is delivered at once and never enters the buffer.

Values flow: `debug snapshot` through Traefik `:9100` shows `Spike`/`RandomInt` Good and updating every second. `GalaxyInt` is Good: it read 11 at 04:27 and 753 at 04:45 (the Galaxy attribute changes slowly). Central's site-x health report shows both connections `Connected`, tag quality 2/2 and 1/1 good, and both cluster nodes online.

### Finding: unreplicated, a site failover brings up an empty node

The first seeded active kill ([log](2026-11-12-leader-auto-down-env2-drill/seeded-unreplicated-site-x-active-kill.log), [metrics](2026-11-12-leader-auto-down-env2-drill/seeded-unreplicated-site-x-active-kill.metrics), [survivor](2026-11-12-leader-auto-down-env2-drill/seeded-unreplicated-survivor-site-x-a.txt)) passed the cluster gate: `DOWNING` at +28 s, survivor active, rejoin 1 s, 10/10. But **data collection did not resume**. The survivor's Deployment Manager loaded only one stale config from an earlier env2 seeding, found no data connections (`Unknown connection: ScadaBridge Site X`) and ran no scripts for the 2 minutes it was watched (`site_data_connections 0/0`, 0 script runs). The 10 messages the victim had buffered stayed in the victim's own SQLite.

This is how the system is designed, not a #116 regression: site-x's `LocalDb` was **unreplicated**, like the rig's site-b and site-c, so deployed configs, artifacts and the S&F buffer exist only on the node that received them. A data-path failover needs the pair replicated. The rig's site-a and the QA pair are replicated.

**Fix (commit `6c7e150a`):** site-x now replicates. Node-a dials `http://scadabridge-env2-site-x-b:8083`, both nodes use the same dev key, and the sizing matches the rig's site-a. Both nodes were recreated together with the soak command from §3 (image and overlay kept). The baseline snapshot then carried node-b's rows to node-a, and node-a, now active, **delivered node-b's 10 stranded messages** (ticks 04:39:03–04:39:21, up to 243 s late, each exactly once).

**A second, smaller finding:** turning replication on for a pair whose stores have diverged has a boot race. Node-a won the active role at boot and loaded its own stale store at 04:42:54. The snapshot that added the other two configs and both connections arrived at 04:42:55. Its startup reconcile at 04:42:59 then found 0 gaps, because the rows were already in SQLite, so the instances never started. A `site deploy-artifacts` plus a redeploy of the two instances fixed it. Steady-state failovers are not affected, because the survivor's singleton loads from an already-converged store. It matters only for a first-time enable, or a node that boots active on a stale store.

### Failovers with data flowing (site-x replicated)

For each site drill, the sink was set to `fail` 20 s before the hit and back to `ok` 5 s after the drill script finished, so messages were buffered on the victim before it died and on the survivor during the takeover. The tick accounting is [`analyze.py`](2026-11-12-leader-auto-down-env2-drill/analyze.py) over the sink's rows. The per-node metrics are `/metrics` on 8084, every 5 s ([`sample.sh`](2026-11-12-leader-auto-down-env2-drill/sample.sh)).

| Run | Cluster gate | Data gap | Buffered messages | Duplicates |
|---|---|---|---|---|
| site-x active **kill** (victim site-x-a) — [log](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-kill.log), [metrics](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-kill.metrics), [ticks](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-kill.ticks) | PASS: `DOWNING` +29 s, survivor active 29 s, rejoin 1 s, 10/10 | **30 s** (04:45:18–04:45:46): kill to the survivor's first `SoakTick` | Victim queued ~11 before the kill, survivor queued 9. Survivor delivered **20**, depth 0 within 10 s of `ok`. Every tick outside the gap present | **0** |
| site-x active **freeze** (`pause`, victim site-x-b) — [log](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-pause.log), [metrics](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-pause.metrics), [ticks](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-active-pause.ticks), [victim](2026-11-12-leader-auto-down-env2-drill/seeded-site-x-pause-victim-lines.txt) | PASS: takeover 27 s; victim `shut out … RESTARTS`, exit 75, rejoined 38 s after unpause, 10/10 | **26 s** (04:48:14–04:48:38) | Victim queued 8 before the freeze and 8 during its zombie window; survivor queued 26. Survivor delivered **42** (= 26 + 16 inherited), depth 0 | **4**, all from the zombie window (below) |
| central active **kill** (victim central-b) with data flowing — [log](2026-11-12-leader-auto-down-env2-drill/seeded-central-active-kill.log), [metrics](2026-11-12-leader-auto-down-env2-drill/seeded-central-active-kill.metrics), [ticks](2026-11-12-leader-auto-down-env2-drill/seeded-central-active-kill.ticks) | PASS: survivor active 27 s, Traefik 29 s, rejoin 3 s, 10/10 | **0**: 43/43 ticks, site collection unaffected | n/a (sink `ok`) | **0** |

**The freeze duplicates are the accepted dual-active trade, not S&F redelivery.** At unpause (04:48:41) the frozen victim still believed it was active for 6 s, until its shut-out at 04:48:47:

- Its Interval timer fired the missed runs as one burst: 4 runs in flight, which is the cap, plus `run shed — 4 runs already in flight (cap 4)`. All 4 carried the same timestamp (`sentAtUtc 04:48:41.727`), so tick 04:48:40 appears 4×.
- One more of its runs (04:48:45.483) overlapped the survivor's own run for tick 04:48:44 (04:48:45.366), so that tick appears 2×.

All of these runs were buffered on the victim, replicated to the survivor, and delivered once each. No message was delivered twice. The #132 shut-out restart bounds the window to about 6 s.

**Whole soak** ([ticks](2026-11-12-leader-auto-down-env2-drill/seeded-whole-soak.ticks), 04:36:30–04:55:01): 400 rows, 396 unique ticks. The only gaps are the three takeover windows: the unreplicated one (264 s, until the replication recreate and redeploy) and 30 s and 26 s for the two replicated drills. The only duplicates are the 4 zombie-window rows. No buffered message was lost.

**Gauge caveat:** the standby's `scadabridge_store_and_forward_queue_depth_items` reports stale values (it showed 10–38 while the standby's copied `sf_messages` table held 0 rows), until the node is reseeded at a restart or a sweep. This is the known approximate gauge (review 2026-10-21 02/C3). Only read it on the active node.

### Final state

([status](2026-11-12-leader-auto-down-env2-drill/seeded-final-status.txt)) All four nodes run `scadabridge:env2-116` with `leader-auto-down`, each `memberCount 2, unreachableCount 0`. The active nodes are **central-a** and **site-x-a**. Traefik `:9100` returns `/health/active` 200. Site-x is replicated, with both connections up and depth 0. `env-soak-1` and `env-galaxy-1` keep running as the soak's live load, and the sink keeps recording at about 1 row per 2 s. Nothing in the primary rig, `scadabridge:latest` or shared infra was changed. `scadabridge-opcua`, `scadabridge-mssql` and the windev gateway were only used as clients.

To keep the soak: leave the sink in `ok`. Running `docker exec scadabridge-env2-sfsink sh -c 'echo fail > /data/mode'` buffers everything until it is set back to `ok`. The sink's `received.jsonl` grows by about 43k rows a day. Use [`run-site.sh`](2026-11-12-leader-auto-down-env2-drill/run-site.sh) to repeat a seeded site drill.