Both centrals killed in turn (Stop-Process on the service PID). SCM restart in ~5 s; the survivor downed the old incarnation at ~+9-10 s and took the active role; the victim rejoined as standby at ~+21 s. leader-auto-down stood aside (already Down), so its 15 s timer path was not exercised on QA. Site-pair drill still pending (Z28061 slot, #142). Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
8.4 KiB
2026-11-15: #116 leader-auto-down G14 central kill drill, QA vq03/vq04
Owner go: 2026-11-15, "yeah" to running the central drill only. WinRM was explicitly authorized for the drill steps ("i authorize winrm"). The site-pair drill was not run: it re-hosts every instance, including Z28061, which another team uses, and no slot has been agreed. It is still pending and goes with #142.
Build: 8e32f353 on both centrals (deployed 2026-11-13, record 2026-11-13-wonder-app-vq03-vq04-8e32f353.md). ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down, with LeaderAutoDown at its defaults (unreachable-after 15 s, shut-out restart ON).
Scope:
- Remote exec (WinRM):
- a preflight of service state and recovery config;
- finding the
ScadaBridge-Centralprocess ID through the service (never by image name, becauseapp\is shared withScadaBridge-Site); Stop-Process -Forceon that process ID only;- a post-kill service-state read.
- Logs: pulled over SMB.
- Health: sampled over HTTP from the Mac.
- Not touched: the
ScadaBridge-Siteservices (process IDs unchanged throughout: vq03 7008, vq04 14332), DEV, and every other app.
Preflight (22:43Z)
| Check | Result |
|---|---|
| Central health | vq03 ready 200 / active 503, vq04 ready 200 / active 200. Exactly one active: vq04 |
| Services | ScadaBridge-Central Running (vq03 process 19652, vq04 15040); ScadaBridge-Site Running (vq03 7008, vq04 14332) |
| Recovery | Both centrals restart after 5 s / 5 s / 10 s, reset after 1 day; FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE |
| CLI | instance list --url http://wonder-app-vq04…:8085 returns 11 instances (QA set) |
vq03's Central process had started at 2026-11-15T13:55:26Z, so the active role moved to vq04 before this drill, not during it.
Timings
Times are UTC, measured from the Stop-Process timestamp. Takeover is the first /health/active 200 on the survivor, sampled about every 1.3 s from the Mac. The other events come from the two Central logs.
| Event | Drill A: kill vq04 (active) | Drill B: kill vq03 (new active) |
|---|---|---|
| Kill | 22:45:19.305, process 15040 | 22:49:24.222, process 19652 |
| SCM starts a new process | 22:45:24.517 (+5.2 s), process 13260 | 22:49:29.347 (+5.1 s), process 27600 |
| Victim cluster node starting | 22:45:28.669 (+9.4 s) | 22:49:32.931 (+8.7 s) |
Survivor: "New incarnation of existing member … is trying to join"; old member marked Down |
22:45:29.222 (+9.9 s) | 22:49:33.223 (+9.0 s) |
Survivor /health/active 200 (takeover) |
22:45:30.525 (+11.2 s); previous sample 22:45:28.992 was 503 | 22:49:33.710 (+9.5 s) |
Victim JOINING → Up (rejoins as standby) |
22:45:39.802 → 22:45:40.299 (+21.0 s) | 22:49:44.010 → 22:49:44.804 (+20.6 s) |
Victim /health/ready 200, /health/active 503 |
by 22:47:37 (sampler gap, below); akka-cluster health errors stop at 22:45:39 |
22:49:45.069 (+20.8 s) |
| Unavailable window (no active central) | ~11 s | ~9.5 s |
Sampler gap, Drill A: the Mac-side sampler stalled from 22:45:37 to 22:47:37. A curl hung past its -m limit, and Drill B ran with --connect-timeout. The victim's Up time comes from the survivor's log (22:45:40.3), and its health-check errors stop at 22:45:39. It was ready 200 / active 503 by the next sample.
Finding: the restart beat the downing timer, so [leader-auto-down] … DOWNING did not fire
The pack expected the survivor to log [leader-auto-down] … DOWNING at about +27 s, followed by a takeover at about +30 s. That is the path when the victim stays away. Here the SCM restarted the killed process after 5 s. Its new incarnation reached the survivor at about +9–10 s, well inside unreachable-after (15 s). Akka core then downed the old incarnation itself ("Existing will be removed from the cluster and then new member will be allowed to join"), and leader-auto-down correctly stood aside:
22:45:29.222 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.3:8081, 552687240)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
22:45:29.227 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:45:29.230 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable but already "Down"; not scheduling a Down
22:45:29.305 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:45:40.299 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.3:8081] to ["Up"]
Drill B (survivor vq04):
22:49:33.223 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.2:8081, 1778435843)] is trying to join. …
22:49:33.228 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.2:8081] as ["Down"]
22:49:33.231 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.2:8081 is unreachable but already "Down"; not scheduling a Down
22:49:33.232 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 gained leadership (leader is now akka.tcp://scadabridge@10.220.158.3:8081); pending unreachable=[]
22:49:44.804 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.2:8081] to ["Up"]
22:49:44.805 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 lost leadership (leader is now akka.tcp://scadabridge@10.220.158.2:8081); pending unreachable=[]
On restart, each victim logged [leader-auto-down] started … unreachable-after="00:00:15" and shut-out restart is ON (G6, again).
What this drill proves: with the QA service recovery config, a crashed or killed central is replaced in about 10 s. There's exactly one active central throughout, and the victim rejoins as standby with no operator action. That is faster than the timer path.
What it does not prove: the leader-auto-down timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). It is proven on the rig and on docker-env2 (#116 checklist), not on QA. To exercise it on QA, the victim must stay away for more than 15 s (Stop-Service, with the SCM not restarting it, or a firewall block), and that is a different drill.
The Akka leader (the lowest address) and the active node (the oldest Up member) differ after a restart, as documented: after Drill B the leader is vq03 and the active node is vq04.
Final gates (22:51Z)
| Gate | Result |
|---|---|
| One active per pair | Central: vq04 active 200, vq03 503. Site (:8084): vq04 active 200, vq03 503 |
| Ready ×4 | Central vq03/vq04 200; Site vq03/vq04 200 |
| Cluster | Central and site akka-cluster Healthy, memberCount 2, unreachable 0 |
| Site LocalDb | localdb Healthy, replicationConnected: true on both site nodes |
| Sites untouched | ScadaBridge-Site process IDs vq03 7008, vq04 14332, the same as the preflight |
| Inbound API | A bogus X-API-Key on POST /api/SimpleAlarmStatusRequest returns 401 on both nodes |
| CLI | instance list against the new active returned 11 instances after each drill |
Overview (vq03:5320) |
ScadaBridge Up, 4 instances, 2 clusters |
End state: vq04 is the active central, vq03 the standby (Central process 27600). vq04's Central runs process 13260, restarted in Drill A. No config was changed.
Still pending for #116
- QA site-pair drill: pending a Z28061 slot, to go with #142, because it re-hosts every instance.
- Optional: a QA drill of the
leader-auto-downtimer path (victim held away for more than 15 s), if the owner wants that on QA rather than relying on the rig and env2 evidence. - DEV vd03 and its sites; flipping the shipped default; docs and register row 70 (unchanged).
Rollback
None needed: no configuration or binary changed. If a node had not recovered, the plan was Start-Service ScadaBridge-Central on the victim. If both nodes had been down-ed, the plan was to restart both centrals together (SBR DownIfAlone self-downs on a cold recreate).