ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill.md

leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)

Date: 2026-11-12 (UTC 04:11–04:27 on 2026-11-13) Scope: docker-env2 only (scadabridge-env2-central-a/b, scadabridge-env2-site-x-a/b, Traefik :9100). Approved by the owner as the next #116 rollout step. The primary rig (docker/), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays auto-down. Build: image scadabridge:env2-116 (sha256:42b177cd4dfd…), built with SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh from b495f429 = origin/main a2e9d718 plus the env2 tooling commit (no code change). That commit was rebased onto a92a418d (docs only) as e8a743de before the push. scadabridge:latest, which the primary rig runs, was not rebuilt or retagged (sha256:4389897ee144… before and after). Strategy under test: ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down on both env2 pairs, with the #132 shut-out restart (default on for that strategy, ShutOutGrace 60 s). StableAfter 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig). Baseline: auto-down on the same build, run first. Raw output: 2026-11-12-leader-auto-down-env2-drill/ (one file per run).

Result

Every #116 gate passes on both env2 pairs. Timings match the primary-rig run (drill doc 2026-11-12-leader-auto-down-rig-drill.md §3 and §7) to within a few seconds.

Gate	env2 central	env2 site-x
Baseline auto-down, active kill	Down +27 s, Traefik 29 s, rejoin 2 s, 10/10	Leader is auto-downing +28 s, rejoin 12 s, 10/10
Baseline boot logs (positive control)	Deprecation warning present on both nodes	Present on both nodes
Kill the active node	PASS. Unreachable +13 s, DOWNING +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10	PASS. DOWNING +27 s, survivor active 27 s, rejoin 1 s, 10/10
Kill the standby	PASS. DOWNING +29 s, 0 /health/active blips (node and Traefik), rejoin 2 s	PASS. DOWNING +28 s, 0 blips, rejoin 1 s
Boot logs	PASS. 0 deprecation lines; [leader-auto-down] started … and shut-out restart is ON … on both nodes; drills show is unreachable; it will be downed if still unreachable after 00:00:15, then DOWNING	PASS, same
Freeze (DRILL_MODE=active DRILL_ACTION=pause)	PASS. Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged shut out by … tiebreak not needed — this node RESTARTS, then exit code 75, and rejoined as standby 3 s after unpause. 10/10	PASS. Takeover 27 s. Victim (site-x-b) … RESTARTS, exit 75, rejoined 8 s after unpause. 10/10
Split past the window, then heal (iptables cut 50 s / 45 s)	PASS. Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a STAY, central-b RESTART; only central-b restarted (+74 s after heal). One active, 2 members at +76 s after heal, 10/10 stable	PASS. Dual-active at +26 s. Healed +46 s. site-x-a STAY, site-x-b RESTART; only site-x-b restarted (+79 s). One active, 2 members at +80 s after heal, 10/10
Kill regression (after the freeze/split runs)	PASS, unchanged. Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node	PASS, unchanged. Down +26 s, rejoin 1 s, 0 shut-out lines

Each rejoin check requires the victim at /health/ready 200 and /health/active 503, and the survivor at memberCount 2, unreachableCount 0; "10/10" is ten one-second samples with exactly one active node.

The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after exit code 75, while the rig's central host took about 30 s. It is not a behaviour difference.

1. Method

• Image isolation. Two committed, default-preserving options (e8a743de):
  • docker/build.sh reads SCADABRIDGE_IMAGE (default scadabridge:latest) for the tag.
  • docker-env2/docker-compose.yml runs ${SCADABRIDGE_IMAGE:-scadabridge:latest} for the four nodes. Unset gives the old behaviour.
• Bind mounts. The compose files came from the worktree, but every run used --project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2, so appsettings and logs stayed bind-mounted from the main checkout, not from the throwaway worktree. env2's appsettings are identical on 7e34d7c2 (main checkout) and a2e9d718.
• Strategy switch. New opt-in overlay docker-env2/docker-compose.leader-auto-down.yml, mirroring the rig's. It is an environment override; the mounted appsettings still say auto-down. Both pairs were switched in one up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
• Drill script. docker/failover-drill.sh gained DRILL_ENV=primary|env2 (container prefix scadabridge-env2-, Traefik :9100, pair site-x). The primary default is unchanged.
• Split and heal. The rig's partition-heal.sh with an optional PREFIX for the container names (copy). It needs the local sb-drill-net:local image (alpine + iptables + curl).

2. Limits of this run

• env2 is not seeded. ScadaBridgeConfig2 has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. docker-env2/seed-sites.sh would bind a data connection to the windev MxGateway (10.100.0.48:5120); it was not run, because that reaches outside env2. Closed 2026-11-13: see §5.
• Transport smoke skipped. docker-env2/README.md points only to 2026-05-24-second-environment-verification.md, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 /management calls at 04:15–04:17). No export or import was run.

3. Final state: env2 stays on leader-auto-down (soak)

Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is left on leader-auto-down as the soak environment for the rollout (final status):

• All four nodes run scadabridge:env2-116 with the overlay env set, all ready, each memberCount 2, unreachableCount 0.
• The active nodes are central-b and site-x-b. Traefik :9100 answers /health/active 200.
• After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.

To keep the soak state when touching env2, always pass both the image and the overlay:

SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
  -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
  up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b

A plain bash docker-env2/deploy.sh rebuilds and runs scadabridge:latest on auto-down, which ends the soak. Rollback is the same command without the overlay file, applied to both pairs together.

4. Next step

QA pair vq03/vq04, which needs owner approval. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with failureflag 1, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.

5. Seeded data-path soak (2026-11-13 UTC 04:32–04:55)

The gap in §2 is closed: env2 now carries real data, and the site-x failovers were re-run with it flowing. Raw output is in the same directory, prefixed seeded-.

What was seeded

• docker-env2/seed-sites.sh, as committed: Site-X (id 3), the Engineering Alerts list, the MxGateway connection ScadaBridge Site X to the windev gateway http://10.100.0.48:5120, the LDAP mappings, and the MxAlarmDemo-1 instance. The connection authenticates with the API key mxgw_scadabridgeshared_…, which is written into the script itself. The management API returns it as ***REDACTED*** with hasSecrets: true. The windev gateway was only used as a client. The LDAP step logged two harmless 2601 duplicate-name errors, because the names already exist under other ids.
• New docker-env2/seed-data-path.sh (commit 8b6d3268):
  • The env2-owned sink container scadabridge-env2-sfsink (docker-env2/sfsink/sink.py). It records every POST, and answers 503 while its mode file says fail.
  • The OPC UA connection OPC PLC Simulator to the shared scadabridge-opcua.
  • The external system EnvSink (method Record, retry every 5 s, retry budget 100000).
  • The instance env-soak-1: Spike from ns=3;s=SpikeData, RandomInt from ns=3;s=RandomSignedInt32, and a 2 s Interval script SoakTick. Each run sends ExternalSystem.CachedCall("EnvSink","Record") with tick = unixSeconds / 2. The sink therefore gets one row per tick: a missing tick is either a run that never happened or a lost message, and a repeated tick is a duplicate.
  • The instance env-galaxy-1: GalaxyInt from TestMachine_003.TestChangingInt, read only (no write binding, no script).
• The sink's mode switch is what makes store-and-forward buffer: a 503 is transient, so the call is parked and retried. A healthy target is delivered at once and never enters the buffer.

Values flow: debug snapshot through Traefik :9100 shows Spike/RandomInt Good and updating every second. GalaxyInt is Good: it read 11 at 04:27 and 753 at 04:45 (the Galaxy attribute changes slowly). Central's site-x health report shows both connections Connected, tag quality 2/2 and 1/1 good, and both cluster nodes online.

Finding: unreplicated, a site failover brings up an empty node

The first seeded active kill (log, metrics, survivor) passed the cluster gate: DOWNING at +28 s, survivor active, rejoin 1 s, 10/10. But data collection did not resume. The survivor's Deployment Manager loaded only one stale config from an earlier env2 seeding, found no data connections (Unknown connection: ScadaBridge Site X) and ran no scripts for the 2 minutes it was watched (site_data_connections 0/0, 0 script runs). The 10 messages the victim had buffered stayed in the victim's own SQLite.

This is how the system is designed, not a #116 regression: site-x's LocalDb was unreplicated, like the rig's site-b and site-c, so deployed configs, artifacts and the S&F buffer exist only on the node that received them. A data-path failover needs the pair replicated. The rig's site-a and the QA pair are replicated.

Fix (commit 6c7e150a): site-x now replicates. Node-a dials http://scadabridge-env2-site-x-b:8083, both nodes use the same dev key, and the sizing matches the rig's site-a. Both nodes were recreated together with the soak command from §3 (image and overlay kept). The baseline snapshot then carried node-b's rows to node-a, and node-a, now active, delivered node-b's 10 stranded messages (ticks 04:39:03–04:39:21, up to 243 s late, each exactly once).

A second, smaller finding: turning replication on for a pair whose stores have diverged has a boot race. Node-a won the active role at boot and loaded its own stale store at 04:42:54. The snapshot that added the other two configs and both connections arrived at 04:42:55. Its startup reconcile at 04:42:59 then found 0 gaps, because the rows were already in SQLite, so the instances never started. A site deploy-artifacts plus a redeploy of the two instances fixed it. Steady-state failovers are not affected, because the survivor's singleton loads from an already-converged store. It matters only for a first-time enable, or a node that boots active on a stale store.

Failovers with data flowing (site-x replicated)

For each site drill, the sink was set to fail 20 s before the hit and back to ok 5 s after the drill script finished, so messages were buffered on the victim before it died and on the survivor during the takeover. The tick accounting is analyze.py over the sink's rows. The per-node metrics are /metrics on 8084, every 5 s (sample.sh).

Run	Cluster gate	Data gap	Buffered messages	Duplicates
site-x active kill (victim site-x-a) — log, metrics, ticks	PASS: DOWNING +29 s, survivor active 29 s, rejoin 1 s, 10/10	30 s (04:45:18–04:45:46): kill to the survivor's first SoakTick	Victim queued ~11 before the kill, survivor queued 9. Survivor delivered 20, depth 0 within 10 s of ok. Every tick outside the gap present	0
site-x active freeze (pause, victim site-x-b) — log, metrics, ticks, victim	PASS: takeover 27 s; victim shut out … RESTARTS, exit 75, rejoined 38 s after unpause, 10/10	26 s (04:48:14–04:48:38)	Victim queued 8 before the freeze and 8 during its zombie window; survivor queued 26. Survivor delivered 42 (= 26 + 16 inherited), depth 0	4, all from the zombie window (below)
central active kill (victim central-b) with data flowing — log, metrics, ticks	PASS: survivor active 27 s, Traefik 29 s, rejoin 3 s, 10/10	0: 43/43 ticks, site collection unaffected	n/a (sink ok)	0

The freeze duplicates are the accepted dual-active trade, not S&F redelivery. At unpause (04:48:41) the frozen victim still believed it was active for 6 s, until its shut-out at 04:48:47:

• Its Interval timer fired the missed runs as one burst: 4 runs in flight, which is the cap, plus run shed — 4 runs already in flight (cap 4). All 4 carried the same timestamp (sentAtUtc 04:48:41.727), so tick 04:48:40 appears 4×.
• One more of its runs (04:48:45.483) overlapped the survivor's own run for tick 04:48:44 (04:48:45.366), so that tick appears 2×.

All of these runs were buffered on the victim, replicated to the survivor, and delivered once each. No message was delivered twice. The #132 shut-out restart bounds the window to about 6 s.

Whole soak (ticks, 04:36:30–04:55:01): 400 rows, 396 unique ticks. The only gaps are the three takeover windows: the unreplicated one (264 s, until the replication recreate and redeploy) and 30 s and 26 s for the two replicated drills. The only duplicates are the 4 zombie-window rows. No buffered message was lost.

Gauge caveat: the standby's scadabridge_store_and_forward_queue_depth_items reports stale values (it showed 10–38 while the standby's copied sf_messages table held 0 rows), until the node is reseeded at a restart or a sweep. This is the known approximate gauge (review 2026-10-21 02/C3). Only read it on the active node.

Final state

(status) All four nodes run scadabridge:env2-116 with leader-auto-down, each memberCount 2, unreachableCount 0. The active nodes are central-a and site-x-a. Traefik :9100 returns /health/active 200. Site-x is replicated, with both connections up and depth 0. env-soak-1 and env-galaxy-1 keep running as the soak's live load, and the sink keeps recording at about 1 row per 2 s. Nothing in the primary rig, scadabridge:latest or shared infra was changed. scadabridge-opcua, scadabridge-mssql and the windev gateway were only used as clients.

To keep the soak: leave the sink in ok. Running docker exec scadabridge-env2-sfsink sh -c 'echo fail > /data/mode' buffers everything until it is set back to ok. The sink's received.jsonl grows by about 43k rows a day. Use run-site.sh to repeat a seeded site drill.