Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill.md
T

175 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# leader-auto-down: docker rig failover drill (ScadaBridge#116)
**Date:** 2026-11-12 (UTC 00:44–01:14 on 2026-11-13)
**Scope:** the local docker rig only (`docker/`, 8 nodes). env2, QA (vq03/vq04) and DEV (vd03) were not touched.
**Build:** `scadabridge:latest` rebuilt from `main` @ `2d2e9f35` with `bash docker/deploy.sh`. This build includes Telemetry.Serilog `2d2e9f35`, LocalDb 0.3.7 `12fa7aff` and OPC SDK `e36cd605`.
**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` (`LeaderAutoDowningProvider`, #106) on the central pair and the site-a pair. `StableAfter` 15 s, heartbeat 2 s, failure-detection threshold 10 s (rig values, unchanged).
**Baseline:** today's default `auto-down` on the same build, run first.
**Raw output:** [`2026-11-12-leader-auto-down-rig-drill/`](2026-11-12-leader-auto-down-rig-drill/) (one file per run).
## Result
**The #116 rig gate passes on both pairs.** Every gate #116 lists (1–4) passed on central and site-a. The timings match the `auto-down` baseline to within about 2 s.
One extra scenario failed for both strategies: a node frozen with `docker pause`. The survivor takes over correctly. But when the frozen node is unpaused, it comes back as a **second active node** and stays that way until it is restarted. `auto-down` does exactly the same thing, so this is parity, not a regression. It is filed as **#132** (§5) and does not block this gate. **Update (same day, §7): fixed by the #132 shut-out restart; the freeze and the healed partition now converge to one active node with no operator action.**
| # | Gate | Central | Site-a |
|---|---|---|---|
| 1/2 | Kill the active (oldest) node: survivor takes over within ~25 s plus hand-over; victim rejoins as standby | **PASS**: Down at +28 s, survivor active at 29 s, Traefik at 30 s, rejoin 2 s | **PASS**: Down at +26 s, survivor active at 26 s, rejoin 11 s |
| 1/2 | Kill the standby: it is downed and removed, with no `/health/active` blips | **PASS**: Down at +29 s, 0 blips, rejoin 2 s | **PASS**: Down at +26 s, 0 blips, rejoin 1 s |
| 3 | Boot logs: no deprecation warning; `[leader-auto-down] started`, then `is unreachable … after 00:00:15`, then `DOWNING` | **PASS** on both nodes | **PASS** on both nodes |
| 4 | Partition held past the window: dual-active happens and both nodes report 200; restarting one side recovers | **PASS**: dual-active at +29 s, still there 60 s after heal, restart recovered in 12 s | **PASS**: dual-active at +27 s, still there 60 s after heal, restart recovered in 32 s |
| extra | Cold start of both pairs together (`docker kill` all four, then `docker start` all four) | **PASS**: one 2-member cluster after 6 s, 0 `DOWNING` | **PASS**: one 2-member cluster after 1 s, 0 `DOWNING` |
| extra | Site keeps collecting and store-and-forward drains across the failover | n/a | **PASS**: the survivor runs scripts and its S&F depth drops to 0, which includes 3 items the dead node left behind |
| extra | Freeze the active node (`docker pause` for about 30 s, then unpause) | Takeover **PASS** (29 s, Traefik 32 s). Victim rejoin **FAIL**: permanent dual-active | Takeover **PASS** (27 s). Victim rejoin **FAIL**: permanent dual-active |
Each "rejoin" check requires all three: the victim's `/health/ready` is 200, its `/health/active` is 503, and the survivor reports `memberCount 2, unreachableCount 0`. After every rejoin, 10 one-second samples each showed exactly one active node.
## 1. Method
- **Config:** a new opt-in overlay, [`docker/docker-compose.leader-auto-down.yml`](../../docker/docker-compose.leader-auto-down.yml). It sets the strategy as an environment override on `central-a/b` and `site-a-a/b`. The mounted appsettings, and the shipped default, still say `auto-down`. Both pairs were switched together and switched back together with `up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b`. There was no rolling restart across the provider change.
- **Drill script:** [`docker/failover-drill.sh`](../../docker/failover-drill.sh) was generalised for this run:
- `DRILL_PAIR=central|site-a|site-b|site-c`. Site `/health/*` is on container port 8084, which is not published, so the script probes from a throwaway sidecar on `scadabridge-net`.
- `DRILL_ACTION=kill|pause`.
- A stricter rejoin check (above), followed by a 10-sample dual-active check.
- The central behaviour and the Traefik check are unchanged.
- **Partition:** iptables `DROP` rules in node B's network namespace, applied from a `NET_ADMIN` sidecar, cutting all traffic between B and A. Nothing else is cut, so both nodes keep SQL, Traefik and the probes. The hold was 50 s on central and 45 s on site-a, longer than the 25 s detection-plus-window. Script: [`partition.sh`](2026-11-12-leader-auto-down-rig-drill/partition.sh).
- **Data and S&F:** `/metrics` on 8084 for each site-a node: `scadabridge_site_script_runs_total`, `site_data_connections_{connected,total}`, `store_and_forward_activity_total{Queued,Delivered}` and `store_and_forward_queue_depth_items`.
- **Overview:** the local `zb-overview` (`:5320`) page, scraped as text.
## 2. Baseline: `auto-down` on the same build
| Run | Result |
|---|---|
| Boot logs, all 4 nodes | The `auto-down-unreachable-after … keep-majority will be used instead` warning is present on every node. This is the **positive control** for gate 3. |
| Central, active kill | Down at +27 s, survivor active at 27 s, Traefik at 30 s, rejoin 3 s, 10/10 single-active ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-kill.log)) |
| Site-a, active kill | `Leader is auto-downing` at +26 s, survivor active at 27 s, rejoin 1 s ([log](2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log)) |
| Central, active pause | Takeover at 29 s, Traefik at 34 s. Then **permanent dual-active**: still both 200 after 120 s. This is the same outcome as `leader-auto-down` ([log](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log)) |
The earlier recorded `auto-down` numbers (2026-07-21, `docker/README.md`) were 28 s for active takeover and 27 s for standby removal.
## 3. `leader-auto-down` runs
### Gate 3: boot logs
After the overlay was applied, there were **0** deprecation lines on `central-a`, `central-b`, `site-a-a` and `site-a-b`. `site-b-a`, still on the default, still logged the warning, which serves as a negative control. Each switched node logged:
```
[00:50:59 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
[00:51:00 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
[00:50:55 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
```
The switch itself was a cold recreate of both pairs at the same moment. Each pair formed one 2-member cluster, with nobody downed.
### Gate 1: central
Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-active-kill.log)). The kill was at 00:51:14:
```
[00:51:27 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
[00:51:27 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
[00:51:42 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
[00:51:43 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
```
The survivor was active at 29 s and Traefik routed to it at 30 s. The victim rejoined as standby 2 s after `docker start`.
Afterwards the survivor logged `lost leadership (leader is now central-a)`. This is expected. The Akka *leader* is the lowest Up address, while the *active* node is the oldest Up member, and central-b stayed active.
Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-standby-kill.log)): unreachable at +14 s, `DOWNING` at +29 s, **0** `/health/active` blips (node and Traefik polled every second), rejoin 2 s.
A second central standby kill was run by mistake with its output discarded, between the two site-a kill runs. It exited without error, and the pair was healthy afterwards. It is not counted.
### Gate 2: site-a
Active kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.log), [metrics](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-kill.metrics)). The kill was at 01:01:56. Unreachable at 01:02:07, `DOWNING` at 01:02:22 (+26 s), `Younger -> Oldest` at 01:02:23, rejoin 11 s.
Data path across the failover:
```
01:01:55 site-a-a script_runs=525 dc_connected=1/2 sf_queued=393 sf_delivered=390 sf_depth=3 <- active before the kill
01:02:46 site-a-b script_runs=17 dc_connected=1/2 sf_queued=12 sf_delivered=15 sf_depth=0 <- survivor, 50 s after the kill
01:02:53 site-a-b script_runs=21 dc_connected=1/2 sf_queued=15 sf_delivered=16
01:04:14 site-a-b script_runs=85 dc_connected=1/2 sf_queued=63 sf_delivered=65
```
The survivor opened the data connection and kept running scripts. It delivered 3 more S&F items than it queued itself, which are the 3 pending items the dead node left in the replicated buffer, so the inherited queue drained to 0.
`dc_connected=1/2` is the rig's steady state before and after: one of site-a's two configured connections never connects on this rig. It is not caused by the drill.
Standby kill ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-standby-kill.log)): `DOWNING` at +26 s, **0** active blips, rejoin 1 s. Collection on the active node carried on throughout (script runs 21 → 85).
### Gate 4: partition
Central ([log](2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log)). Both sides saw the other unreachable at +12 s, and **both** logged `DOWNING` of the other at +27 s. Both were 200 from +29 s. After the cut was removed at +50 s, both were still 200 at every 10 s sample for 60 s, because each side has quarantined the other.
During the dual-active, Overview showed `Split brain · …central-a:8081 vs …central-b:8081`, and each node reported `members 1`. Restarting `central-b` brought back one 2-member cluster with central-a active in **12 s**.
Site-a ([log](2026-11-12-leader-auto-down-rig-drill/lad-site-a-partition.log)): the same sequence, with dual-active from +27 s and Overview showing `Split brain`. Both site nodes had the data connection open and were running scripts, so the data was collected twice. That is the accepted trade. Restarting `site-a-b` recovered in **32 s**.
This matches plan §6 and the in-process `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp`.
### Cold start of both pairs together
[Log](2026-11-12-leader-auto-down-rig-drill/lad-cold-restart-both-pairs.log). All four nodes were killed, then all four started. One 2-member cluster formed in 1 s on site-a and 6 s on central, with exactly one active node per pair and 0 `DOWNING` lines. The same result came from the `--force-recreate` switches at 00:50:42 and 01:01:04.
## 4. Rig state afterwards
Both pairs were recreated together on the default (no overlay) at about 01:14. The rig is on `auto-down` again, and the deprecation warning is back on all four nodes, which is expected for the default.
All 8 nodes are Up. Each pair has one active node (200) and one standby (503): central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status.txt)).
## 5. Finding: a frozen node comes back as a permanent second active (#132)
Runs: [central LAD](2026-11-12-leader-auto-down-rig-drill/lad-central-active-pause.log), [site-a LAD](2026-11-12-leader-auto-down-rig-drill/lad-site-a-active-pause.log), [central auto-down baseline](2026-11-12-leader-auto-down-rig-drill/baseline-central-active-pause.log).
After `docker unpause`, the frozen node (central-b in the first run) logged:
```
[00:53:22 WRN] … Scheduled sending of heartbeat was delayed. Previous heartbeat was sent [33472.866] ms ago …
[00:53:22 WRN] … [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 …
[00:53:28 ERR] … AssociationError … Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.]
[00:53:37 WRN] … [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (…central-b:8081) is leader
```
After that, both nodes reported `/health/active` 200 until the zombie was restarted. That took 180 s on central and 90 s on site-a, both times when the drill script gave up.
The cause is in Akka. Akka.Remote 1.5.62 publishes `ThisActorSystemQuarantinedEvent` (`EndpointManager`), but Akka.Cluster 1.5.62's `ClusterCoreDaemon` subscribes only to the opposite `QuarantinedEvent`. Nothing tells a node that it has been removed. The `auto-down` baseline behaves identically, so `LeaderAutoDowningProvider` is a faithful port in this respect too. It was **not** changed.
The fix (self-down on `ThisActorSystemQuarantinedEvent`) would also make both sides of a healed partition restart. That changes gate 4's contract, so it is left for an owner decision in #132.
## 6. Next step
The next rollout step is **`docker-env2`**, and it **needs owner approval**. Apply the same strategy to its central pair and one site pair, stopping and starting each pair together, and re-run the drill there. After that come the QA pair vq03/vq04, then DEV vd03 and its sites. The default flip and the docs and register row 70 updates wait until the rollout is complete (#116 "Rollout after the gates pass").
Decide #132 before the default flips. **Decided and done — see §7.**
## 7. Re-run with the #132 shut-out restart (2026-11-12, UTC 02:11–02:25 on 2026-11-13)
**Build:** `scadabridge:latest` built with `docker/build.sh` from a worktree at `db8d9ffe` (the #132 fix on `main` `9b0720af`; rebased onto `a633005f` as `952db02a` before the push, with no conflicts and no code change). The four pair nodes were then recreated with the **main checkout's** compose plus `docker-compose.leader-auto-down.yml`, so the bind-mounted appsettings stayed the main checkout's. Both pairs switched together. Boot logs on all four showed `[leader-auto-down] shut-out restart is ON for …` next to `[leader-auto-down] started on …`.
**Fix under test** (`ShutOutRestart`, ScadaBridge#132; design in `Component-ClusterInfrastructure.md` → *Shut-out restart*): a node shut out by its peer restarts itself. If both sides are shut out, the **higher address** restarts after a 60 s grace and the lower stays. The restart uses the self-down path, and the process now exits 75.
| Gate | Central | Site-a |
|---|---|---|
| **Freeze** (`DRILL_MODE=active DRILL_ACTION=pause`): survivor takes over; after unpause, exactly one active node and the victim rejoins as standby, with no operator action | **PASS.** Takeover 27 s, Traefik 31 s. After unpause the victim logged `shut out by central-b … tiebreak not needed — this node RESTARTS` within 1 s (02:15:08), exited 75 and rejoined as standby **34 s** after unpause; 30 s of that was the central host's own shutdown. 10/10 single-active. The survivor logged `split from central-a … Giving it 00:01:00`, then `rejoined (Up) within the shut-out grace; this node stays` ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-active-pause.log)) | **PASS.** Takeover 27 s. Victim `shut out … RESTARTS` at unpause and rejoined **6 s** after unpause. 10/10 single-active ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-active-pause.log)) |
| **Network split, then heal** (iptables cut 50 s / 45 s, past the 25 s window): during the cut the dual-active still happens; after the heal exactly one node restarts and the pair ends with one active node | **PASS.** Dual-active at +27 s (expected while cut). Healed at +51 s; both sides logged `split from …` 12 s after the heal, which was the first probe through. 60 s later both logged the tiebreak: `central-a … Tiebreak says STAY`, `central-b … Tiebreak says RESTART`. **Only central-b restarted** (+74 s after heal, RestartCount 0→1, central-a unchanged). One 2-member cluster with central-a active at **+76 s after heal**, 10/10 stable ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-partition-heal.log)) | **PASS.** Dual-active at +28 s. Healed at +46 s; `split from` +17 s after; tiebreak +77 s: site-a-a STAY, site-a-b RESTART. **Only site-a-b restarted**. One active node (site-a-a), 2 members at **+111 s after heal**; 30 s of that was site-a-b's host shutdown. 10/10 stable. site-a-a later logged `rejoined (Up) within the shut-out grace; this node stays` ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-partition-heal.log)) |
| **Kill regression** (`DRILL_MODE=active`, `docker kill`) | **PASS, unchanged.** Down +26 s, Traefik 28 s, rejoin 2 s, 10/10 single-active. **No** shut-out lines on either node: the victim came back as a new incarnation ([log](2026-11-12-leader-auto-down-rig-drill/sor-central-active-kill.log)) | **PASS, unchanged.** Down +28 s, rejoin 1 s, no shut-out lines ([log](2026-11-12-leader-auto-down-rig-drill/sor-site-a-active-kill.log)) |
For comparison, §3 measured a kill takeover of 29 s / 26 s, and §5 recorded a permanent dual-active after a freeze or a healed partition.
**What the rig confirmed about the signals.** In the freeze case the victim gets S1 (`ThisActorSystemQuarantinedEvent`, logged as `The remote system has quarantined this system`) while it still counts the survivor as Up, so it restarts at once. The survivor gets S2 (`has a UID that has been quarantined`) when its probe reaches the old incarnation, and waits. In the healed partition **neither** side gets S1; both get S2 from their probes, and the tiebreak decides. This matches the in-process `ShutOutRestartScenarioTests`.
**Gate 4 contract change.** The dual-active still appears *during* a partition longer than the window, which is the accepted trade. After the heal it now clears itself in about one probe interval plus the 60 s grace plus one restart. Previously it waited for an operator.
**Timings to expect.** Freeze: one restart of the frozen node, in seconds, plus its host's shutdown time (central's host took about 30 s to stop here; sites took less than 1 s). Healed partition: probe back-off (5 → 60 s, so up to 60 s after a long cut) + 60 s grace + one restart of the higher-address node.
**Rig state afterwards.** All 8 nodes were recreated on the **default `auto-down`** with no overlay, running image `db8d9ffe`. All 8 are ready. Each pair has one active node and `memberCount 2, unreachableCount 0`: central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status-132.txt)). The deprecation warning is back on the default nodes, and no `shut-out restart` line appears, because the actor exists only under `leader-auto-down`.
Helper for the split gate: [`partition-heal.sh`](2026-11-12-leader-auto-down-rig-drill/partition-heal.sh). It is an evidence copy, not a supported rig script.
## 8. docker-env2 (2026-11-12)
The owner approved the next rollout step. All gates, including the freeze and split-then-heal gates, passed on env2's central and site-x pairs, and env2 stays on `leader-auto-down` as the soak environment. Record: [`2026-11-12-leader-auto-down-env2-drill.md`](2026-11-12-leader-auto-down-env2-drill.md).