ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill.md

leader-auto-down: docker rig failover drill (ScadaBridge#116)

Date: 2026-11-12 (UTC 00:44–01:14 on 2026-11-13) Scope: the local docker rig only (docker/, 8 nodes). env2, QA (vq03/vq04) and DEV (vd03) were not touched. Build: scadabridge:latest rebuilt from main @ 2d2e9f35 with bash docker/deploy.sh. This build includes Telemetry.Serilog 2d2e9f35, LocalDb 0.3.7 12fa7aff and OPC SDK e36cd605. Strategy under test: ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down (LeaderAutoDowningProvider, #106) on the central pair and the site-a pair. StableAfter 15 s, heartbeat 2 s, failure-detection threshold 10 s (rig values, unchanged). Baseline: today's default auto-down on the same build, run first. Raw output: 2026-11-12-leader-auto-down-rig-drill/ (one file per run).

Result

The #116 rig gate passes on both pairs. Every gate #116 lists (1–4) passed on central and site-a. The timings match the auto-down baseline to within about 2 s.

One extra scenario failed for both strategies: a node frozen with docker pause. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a second active node and stays that way until it is restarted. auto-down does exactly the same thing, so this is parity, not a regression. It is filed as #132 (§5) and does not block this gate.

#	Gate	Central	Site-a
1/2	Kill the active (oldest) node: survivor takes over within ~25 s plus hand-over; victim rejoins as standby	PASS: Down at +28 s, survivor active at 29 s, Traefik at 30 s, rejoin 2 s	PASS: Down at +26 s, survivor active at 26 s, rejoin 11 s
1/2	Kill the standby: it is downed and removed, with no /health/active blips	PASS: Down at +29 s, 0 blips, rejoin 2 s	PASS: Down at +26 s, 0 blips, rejoin 1 s
3	Boot logs: no deprecation warning; [leader-auto-down] started, then is unreachable … after 00:00:15, then DOWNING	PASS on both nodes	PASS on both nodes
4	Partition held past the window: dual-active happens and both nodes report 200; restarting one side recovers	PASS: dual-active at +29 s, still there 60 s after heal, restart recovered in 12 s	PASS: dual-active at +27 s, still there 60 s after heal, restart recovered in 32 s
extra	Cold start of both pairs together (docker kill all four, then docker start all four)	PASS: one 2-member cluster after 6 s, 0 DOWNING	PASS: one 2-member cluster after 1 s, 0 DOWNING
extra	Site keeps collecting and store-and-forward drains across the failover	n/a	PASS: the survivor runs scripts and its S&F depth drops to 0, which includes 3 items the dead node left behind
extra	Freeze the active node (docker pause for about 30 s, then unpause)	Takeover PASS (29 s, Traefik 32 s). Victim rejoin FAIL: permanent dual-active	Takeover PASS (27 s). Victim rejoin FAIL: permanent dual-active

Each "rejoin" check requires all three: the victim's /health/ready is 200, its /health/active is 503, and the survivor reports memberCount 2, unreachableCount 0. After every rejoin, 10 one-second samples each showed exactly one active node.

1. Method

• Config: a new opt-in overlay, docker/docker-compose.leader-auto-down.yml. It sets the strategy as an environment override on central-a/b and site-a-a/b. The mounted appsettings, and the shipped default, still say auto-down. Both pairs were switched together and switched back together with up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b. There was no rolling restart across the provider change.
• Drill script: docker/failover-drill.sh was generalised for this run:
  • DRILL_PAIR=central|site-a|site-b|site-c. Site /health/* is on container port 8084, which is not published, so the script probes from a throwaway sidecar on scadabridge-net.
  • DRILL_ACTION=kill|pause.
  • A stricter rejoin check (above), followed by a 10-sample dual-active check.
  • The central behaviour and the Traefik check are unchanged.
• Partition: iptables DROP rules in node B's network namespace, applied from a NET_ADMIN sidecar, cutting all traffic between B and A. Nothing else is cut, so both nodes keep SQL, Traefik and the probes. The hold was 50 s on central and 45 s on site-a, longer than the 25 s detection-plus-window. Script: partition.sh.
• Data and S&F: /metrics on 8084 for each site-a node: scadabridge_site_script_runs_total, site_data_connections_{connected,total}, store_and_forward_activity_total{Queued,Delivered} and store_and_forward_queue_depth_items.
• Overview: the local zb-overview (:5320) page, scraped as text.

2. Baseline: auto-down on the same build

Run	Result
Boot logs, all 4 nodes	The auto-down-unreachable-after … keep-majority will be used instead warning is present on every node. This is the positive control for gate 3.
Central, active kill	Down at +27 s, survivor active at 27 s, Traefik at 30 s, rejoin 3 s, 10/10 single-active (log)
Site-a, active kill	Leader is auto-downing at +26 s, survivor active at 27 s, rejoin 1 s (log)
Central, active pause	Takeover at 29 s, Traefik at 34 s. Then permanent dual-active: still both 200 after 120 s. This is the same outcome as leader-auto-down (log)

The earlier recorded auto-down numbers (2026-07-21, docker/README.md) were 28 s for active takeover and 27 s for standby removal.

3. leader-auto-down runs

Gate 3: boot logs

After the overlay was applied, there were 0 deprecation lines on central-a, central-b, site-a-a and site-a-b. site-b-a, still on the default, still logged the warning, which serves as a negative control. Each switched node logged:

[00:50:59 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
[00:51:00 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
[00:50:55 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]

The switch itself was a cold recreate of both pairs at the same moment. Each pair formed one 2-member cluster, with nobody downed.

Gate 1: central

Active kill (log). The kill was at 00:51:14:

[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData

The survivor was active at 29 s and Traefik routed to it at 30 s. The victim rejoined as standby 2 s after docker start.

Afterwards the survivor logged lost leadership (leader is now central-a). This is expected. The Akka leader is the lowest Up address, while the active node is the oldest Up member, and central-b stayed active.

Standby kill (log): unreachable at +14 s, DOWNING at +29 s, 0 /health/active blips (node and Traefik polled every second), rejoin 2 s.

A second central standby kill was run by mistake with its output discarded, between the two site-a kill runs. It exited without error, and the pair was healthy afterwards. It is not counted.

Gate 2: site-a

Active kill (log, metrics). The kill was at 01:01:56. Unreachable at 01:02:07, DOWNING at 01:02:22 (+26 s), Younger -> Oldest at 01:02:23, rejoin 11 s.

Data path across the failover:

01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3   <- active before the kill
01:02:46 site-a-b script_runs=17  dc_connected=1/2 sf_queued=12  sf_delivered=15  sf_depth=0   <- survivor, 50 s after the kill
01:02:53 site-a-b script_runs=21  dc_connected=1/2 sf_queued=15  sf_delivered=16
01:04:14 site-a-b script_runs=85  dc_connected=1/2 sf_queued=63  sf_delivered=65

The survivor opened the data connection and kept running scripts. It delivered 3 more S&F items than it queued itself, which are the 3 pending items the dead node left in the replicated buffer, so the inherited queue drained to 0.

dc_connected=1/2 is the rig's steady state before and after: one of site-a's two configured connections never connects on this rig. It is not caused by the drill.

Standby kill (log): DOWNING at +26 s, 0 active blips, rejoin 1 s. Collection on the active node carried on throughout (script runs 21 → 85).

Gate 4: partition

Central (log). Both sides saw the other unreachable at +12 s, and both logged DOWNING of the other at +27 s. Both were 200 from +29 s. After the cut was removed at +50 s, both were still 200 at every 10 s sample for 60 s, because each side has quarantined the other.

During the dual-active, Overview showed Split brain · …central-a:8081 vs …central-b:8081, and each node reported members 1. Restarting central-b brought back one 2-member cluster with central-a active in 12 s.

Site-a (log): the same sequence, with dual-active from +27 s and Overview showing Split brain. Both site nodes had the data connection open and were running scripts, so the data was collected twice. That is the accepted trade. Restarting site-a-b recovered in 32 s.

This matches plan §6 and the in-process LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp.

Cold start of both pairs together

Log. All four nodes were killed, then all four started. One 2-member cluster formed in 1 s on site-a and 6 s on central, with exactly one active node per pair and 0 DOWNING lines. The same result came from the --force-recreate switches at 00:50:42 and 01:01:04.

4. Rig state afterwards

Both pairs were recreated together on the default (no overlay) at about 01:14. The rig is on auto-down again, and the deprecation warning is back on all four nodes, which is expected for the default.

All 8 nodes are Up. Each pair has one active node (200) and one standby (503): central-a, site-a-a, site-b-a and site-c-a are active (final status).

5. Finding: a frozen node comes back as a permanent second active (#132)

Runs: central LAD, site-a LAD, central auto-down baseline.

After docker unpause, the frozen node (central-b in the first run) logged:

[00:53:22 WRN] … Scheduled sending of heartbeat was delayed. Previous heartbeat was sent [33472.866] ms ago …
[00:53:22 WRN] … [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 …
[00:53:28 ERR] … AssociationError … Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
[00:53:37 WRN] … [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (…central-b:8081) is leader

After that, both nodes reported /health/active 200 until the zombie was restarted. That took 180 s on central and 90 s on site-a, both times when the drill script gave up.

The cause is in Akka. Akka.Remote 1.5.62 publishes ThisActorSystemQuarantinedEvent (EndpointManager), but Akka.Cluster 1.5.62's ClusterCoreDaemon subscribes only to the opposite QuarantinedEvent. Nothing tells a node that it has been removed. The auto-down baseline behaves identically, so LeaderAutoDowningProvider is a faithful port in this respect too. It was not changed.

The fix (self-down on ThisActorSystemQuarantinedEvent) would also make both sides of a healed partition restart. That changes gate 4's contract, so it is left for an owner decision in #132.

6. Next step

The next rollout step is docker-env2, and it needs owner approval. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass").

Decide #132 before the default flips.