ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill.md

leader-auto-down: docker rig failover drill (ScadaBridge#116)

Date: 2026-11-12 (UTC 00:44–01:14 on 2026-11-13) Scope: the local docker rig only (docker/, 8 nodes). env2, QA (vq03/vq04) and DEV (vd03) were not touched. Build: scadabridge:latest rebuilt from main @ 2d2e9f35 with bash docker/deploy.sh. This build includes Telemetry.Serilog 2d2e9f35, LocalDb 0.3.7 12fa7aff and OPC SDK e36cd605. Strategy under test: ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down (LeaderAutoDowningProvider, #106) on the central pair and the site-a pair. StableAfter 15 s, heartbeat 2 s, failure-detection threshold 10 s (rig values, unchanged). Baseline: today's default auto-down on the same build, run first. Raw output: 2026-11-12-leader-auto-down-rig-drill/ (one file per run).

Result

The #116 rig gate passes on both pairs. Every gate #116 lists (1–4) passed on central and site-a. The timings match the auto-down baseline to within about 2 s.

One extra scenario failed for both strategies: a node frozen with docker pause. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a second active node and stays that way until it is restarted. auto-down does exactly the same thing, so this is parity, not a regression. It is filed as #132 (§5) and does not block this gate. Update (same day, §7): fixed by the #132 shut-out restart; the freeze and the healed partition now converge to one active node with no operator action.

#	Gate	Central	Site-a
1/2	Kill the active (oldest) node: survivor takes over within ~25 s plus hand-over; victim rejoins as standby	PASS: Down at +28 s, survivor active at 29 s, Traefik at 30 s, rejoin 2 s	PASS: Down at +26 s, survivor active at 26 s, rejoin 11 s
1/2	Kill the standby: it is downed and removed, with no /health/active blips	PASS: Down at +29 s, 0 blips, rejoin 2 s	PASS: Down at +26 s, 0 blips, rejoin 1 s
3	Boot logs: no deprecation warning; [leader-auto-down] started, then is unreachable … after 00:00:15, then DOWNING	PASS on both nodes	PASS on both nodes
4	Partition held past the window: dual-active happens and both nodes report 200; restarting one side recovers	PASS: dual-active at +29 s, still there 60 s after heal, restart recovered in 12 s	PASS: dual-active at +27 s, still there 60 s after heal, restart recovered in 32 s
extra	Cold start of both pairs together (docker kill all four, then docker start all four)	PASS: one 2-member cluster after 6 s, 0 DOWNING	PASS: one 2-member cluster after 1 s, 0 DOWNING
extra	Site keeps collecting and store-and-forward drains across the failover	n/a	PASS: the survivor runs scripts and its S&F depth drops to 0, which includes 3 items the dead node left behind
extra	Freeze the active node (docker pause for about 30 s, then unpause)	Takeover PASS (29 s, Traefik 32 s). Victim rejoin FAIL: permanent dual-active	Takeover PASS (27 s). Victim rejoin FAIL: permanent dual-active

Each "rejoin" check requires all three: the victim's /health/ready is 200, its /health/active is 503, and the survivor reports memberCount 2, unreachableCount 0. After every rejoin, 10 one-second samples each showed exactly one active node.

1. Method

• Config: a new opt-in overlay, docker/docker-compose.leader-auto-down.yml. It sets the strategy as an environment override on central-a/b and site-a-a/b. The mounted appsettings, and the shipped default, still say auto-down. Both pairs were switched together and switched back together with up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b. There was no rolling restart across the provider change.
• Drill script: docker/failover-drill.sh was generalised for this run:
  • DRILL_PAIR=central|site-a|site-b|site-c. Site /health/* is on container port 8084, which is not published, so the script probes from a throwaway sidecar on scadabridge-net.
  • DRILL_ACTION=kill|pause.
  • A stricter rejoin check (above), followed by a 10-sample dual-active check.
  • The central behaviour and the Traefik check are unchanged.
• Partition: iptables DROP rules in node B's network namespace, applied from a NET_ADMIN sidecar, cutting all traffic between B and A. Nothing else is cut, so both nodes keep SQL, Traefik and the probes. The hold was 50 s on central and 45 s on site-a, longer than the 25 s detection-plus-window. Script: partition.sh.
• Data and S&F: /metrics on 8084 for each site-a node: scadabridge_site_script_runs_total, site_data_connections_{connected,total}, store_and_forward_activity_total{Queued,Delivered} and store_and_forward_queue_depth_items.
• Overview: the local zb-overview (:5320) page, scraped as text.

2. Baseline: auto-down on the same build

Run	Result
Boot logs, all 4 nodes	The auto-down-unreachable-after … keep-majority will be used instead warning is present on every node. This is the positive control for gate 3.
Central, active kill	Down at +27 s, survivor active at 27 s, Traefik at 30 s, rejoin 3 s, 10/10 single-active (log)
Site-a, active kill	Leader is auto-downing at +26 s, survivor active at 27 s, rejoin 1 s (log)
Central, active pause	Takeover at 29 s, Traefik at 34 s. Then permanent dual-active: still both 200 after 120 s. This is the same outcome as leader-auto-down (log)

The earlier recorded auto-down numbers (2026-07-21, docker/README.md) were 28 s for active takeover and 27 s for standby removal.

3. leader-auto-down runs

Gate 3: boot logs

After the overlay was applied, there were 0 deprecation lines on central-a, central-b, site-a-a and site-a-b. site-b-a, still on the default, still logged the warning, which serves as a negative control. Each switched node logged:

[00:50:59 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
[00:51:00 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
[00:50:55 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]

The switch itself was a cold recreate of both pairs at the same moment. Each pair formed one 2-member cluster, with nobody downed.

Gate 1: central

Active kill (log). The kill was at 00:51:14:

[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData

The survivor was active at 29 s and Traefik routed to it at 30 s. The victim rejoined as standby 2 s after docker start.

Afterwards the survivor logged lost leadership (leader is now central-a). This is expected. The Akka leader is the lowest Up address, while the active node is the oldest Up member, and central-b stayed active.

Standby kill (log): unreachable at +14 s, DOWNING at +29 s, 0 /health/active blips (node and Traefik polled every second), rejoin 2 s.

A second central standby kill was run by mistake with its output discarded, between the two site-a kill runs. It exited without error, and the pair was healthy afterwards. It is not counted.

Gate 2: site-a

Active kill (log, metrics). The kill was at 01:01:56. Unreachable at 01:02:07, DOWNING at 01:02:22 (+26 s), Younger -> Oldest at 01:02:23, rejoin 11 s.

Data path across the failover:

01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3   <- active before the kill
01:02:46 site-a-b script_runs=17  dc_connected=1/2 sf_queued=12  sf_delivered=15  sf_depth=0   <- survivor, 50 s after the kill
01:02:53 site-a-b script_runs=21  dc_connected=1/2 sf_queued=15  sf_delivered=16
01:04:14 site-a-b script_runs=85  dc_connected=1/2 sf_queued=63  sf_delivered=65

The survivor opened the data connection and kept running scripts. It delivered 3 more S&F items than it queued itself, which are the 3 pending items the dead node left in the replicated buffer, so the inherited queue drained to 0.

dc_connected=1/2 is the rig's steady state before and after: one of site-a's two configured connections never connects on this rig. It is not caused by the drill.

Standby kill (log): DOWNING at +26 s, 0 active blips, rejoin 1 s. Collection on the active node carried on throughout (script runs 21 → 85).

Gate 4: partition

Central (log). Both sides saw the other unreachable at +12 s, and both logged DOWNING of the other at +27 s. Both were 200 from +29 s. After the cut was removed at +50 s, both were still 200 at every 10 s sample for 60 s, because each side has quarantined the other.

During the dual-active, Overview showed Split brain · …central-a:8081 vs …central-b:8081, and each node reported members 1. Restarting central-b brought back one 2-member cluster with central-a active in 12 s.

Site-a (log): the same sequence, with dual-active from +27 s and Overview showing Split brain. Both site nodes had the data connection open and were running scripts, so the data was collected twice. That is the accepted trade. Restarting site-a-b recovered in 32 s.

This matches plan §6 and the in-process LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp.

Cold start of both pairs together

Log. All four nodes were killed, then all four started. One 2-member cluster formed in 1 s on site-a and 6 s on central, with exactly one active node per pair and 0 DOWNING lines. The same result came from the --force-recreate switches at 00:50:42 and 01:01:04.

4. Rig state afterwards

Both pairs were recreated together on the default (no overlay) at about 01:14. The rig is on auto-down again, and the deprecation warning is back on all four nodes, which is expected for the default.

All 8 nodes are Up. Each pair has one active node (200) and one standby (503): central-a, site-a-a, site-b-a and site-c-a are active (final status).

5. Finding: a frozen node comes back as a permanent second active (#132)

Runs: central LAD, site-a LAD, central auto-down baseline.

After docker unpause, the frozen node (central-b in the first run) logged:

[00:53:22 WRN] … Scheduled sending of heartbeat was delayed. Previous heartbeat was sent [33472.866] ms ago …
[00:53:22 WRN] … [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 …
[00:53:28 ERR] … AssociationError … Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
[00:53:37 WRN] … [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (…central-b:8081) is leader

After that, both nodes reported /health/active 200 until the zombie was restarted. That took 180 s on central and 90 s on site-a, both times when the drill script gave up.

The cause is in Akka. Akka.Remote 1.5.62 publishes ThisActorSystemQuarantinedEvent (EndpointManager), but Akka.Cluster 1.5.62's ClusterCoreDaemon subscribes only to the opposite QuarantinedEvent. Nothing tells a node that it has been removed. The auto-down baseline behaves identically, so LeaderAutoDowningProvider is a faithful port in this respect too. It was not changed.

The fix (self-down on ThisActorSystemQuarantinedEvent) would also make both sides of a healed partition restart. That changes gate 4's contract, so it is left for an owner decision in #132.

6. Next step

The next rollout step is docker-env2, and it needs owner approval. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass").

Decide #132 before the default flips. Decided and done — see §7.

7. Re-run with the #132 shut-out restart (2026-11-12, UTC 02:11–02:25 on 2026-11-13)

Build: scadabridge:latest built with docker/build.sh from a worktree at db8d9ffe (the #132 fix on main 9b0720af; rebased onto a633005f as 952db02a before the push, with no conflicts and no code change). The four pair nodes were then recreated with the main checkout's compose plus docker-compose.leader-auto-down.yml, so the bind-mounted appsettings stayed the main checkout's. Both pairs switched together. Boot logs on all four showed [leader-auto-down] shut-out restart is ON for … next to [leader-auto-down] started on ….

Fix under test (ShutOutRestart, ScadaBridge#132; design in Component-ClusterInfrastructure.md → Shut-out restart): a node shut out by its peer restarts itself. If both sides are shut out, the higher address restarts after a 60 s grace and the lower stays. The restart uses the self-down path, and the process now exits 75.

Gate	Central	Site-a
Freeze (DRILL_MODE=active DRILL_ACTION=pause): survivor takes over; after unpause, exactly one active node and the victim rejoins as standby, with no operator action	PASS. Takeover 27 s, Traefik 31 s. After unpause the victim logged shut out by central-b … tiebreak not needed — this node RESTARTS within 1 s (02:15:08), exited 75 and rejoined as standby 34 s after unpause; 30 s of that was the central host's own shutdown. 10/10 single-active. The survivor logged split from central-a … Giving it 00:01:00, then rejoined (Up) within the shut-out grace; this node stays (log)	PASS. Takeover 27 s. Victim shut out … RESTARTS at unpause and rejoined 6 s after unpause. 10/10 single-active (log)
Network split, then heal (iptables cut 50 s / 45 s, past the 25 s window): during the cut the dual-active still happens; after the heal exactly one node restarts and the pair ends with one active node	PASS. Dual-active at +27 s (expected while cut). Healed at +51 s; both sides logged split from … 12 s after the heal, which was the first probe through. 60 s later both logged the tiebreak: central-a … Tiebreak says STAY, central-b … Tiebreak says RESTART. Only central-b restarted (+74 s after heal, RestartCount 0→1, central-a unchanged). One 2-member cluster with central-a active at +76 s after heal, 10/10 stable (log)	PASS. Dual-active at +28 s. Healed at +46 s; split from +17 s after; tiebreak +77 s: site-a-a STAY, site-a-b RESTART. Only site-a-b restarted. One active node (site-a-a), 2 members at +111 s after heal; 30 s of that was site-a-b's host shutdown. 10/10 stable. site-a-a later logged rejoined (Up) within the shut-out grace; this node stays (log)
Kill regression (DRILL_MODE=active, docker kill)	PASS, unchanged. Down +26 s, Traefik 28 s, rejoin 2 s, 10/10 single-active. No shut-out lines on either node: the victim came back as a new incarnation (log)	PASS, unchanged. Down +28 s, rejoin 1 s, no shut-out lines (log)

For comparison, §3 measured a kill takeover of 29 s / 26 s, and §5 recorded a permanent dual-active after a freeze or a healed partition.

What the rig confirmed about the signals. In the freeze case the victim gets S1 (ThisActorSystemQuarantinedEvent, logged as The remote system has quarantined this system) while it still counts the survivor as Up, so it restarts at once. The survivor gets S2 (has a UID that has been quarantined) when its probe reaches the old incarnation, and waits. In the healed partition neither side gets S1; both get S2 from their probes, and the tiebreak decides. This matches the in-process ShutOutRestartScenarioTests.

Gate 4 contract change. The dual-active still appears during a partition longer than the window, which is the accepted trade. After the heal it now clears itself in about one probe interval plus the 60 s grace plus one restart. Previously it waited for an operator.

Timings to expect. Freeze: one restart of the frozen node, in seconds, plus its host's shutdown time (central's host took about 30 s to stop here; sites took less than 1 s). Healed partition: probe back-off (5 → 60 s, so up to 60 s after a long cut) + 60 s grace + one restart of the higher-address node.

Rig state afterwards. All 8 nodes were recreated on the default auto-down with no overlay, running image db8d9ffe. All 8 are ready. Each pair has one active node and memberCount 2, unreachableCount 0: central-a, site-a-a, site-b-a and site-c-a are active (final status). The deprecation warning is back on the default nodes, and no shut-out restart line appears, because the actor exists only under leader-auto-down.

Helper for the split gate: partition-heal.sh. It is an evidence copy, not a supported rig script.

8. docker-env2 (2026-11-12)

The owner approved the next rollout step. All gates, including the freeze and split-then-heal gates, passed on env2's central and site-x pairs, and env2 stays on leader-auto-down as the soak environment. Record: 2026-11-12-leader-auto-down-env2-drill.md.