vq04 (active central) killed with SCM auto-restart disabled and held down 94 s: UNREACHABLE +11.4 s, [leader-auto-down] DOWNING +26.4 s, survivor vq03 active +26.7 s. Recovery config restored byte-identical before Start-Service; vq04 rejoined as standby in ~4.7 s with no exit-75. Also corrects the earlier Drill A 'sampler gap' (an awk keying error; vq04 was ready at +20.8 s). Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
14 KiB
2026-11-15: #116 leader-auto-down G14 central kill drill, QA vq03/vq04
Owner go: 2026-11-15, "yeah" to running the central drill only. WinRM was explicitly authorized for the drill steps ("i authorize winrm"). The site-pair drill was not run: it re-hosts every instance, including Z28061, which another team uses, and no slot has been agreed. It is still pending and goes with #142.
Build: 8e32f353 on both centrals (deployed 2026-11-13, record 2026-11-13-wonder-app-vq03-vq04-8e32f353.md). ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down, with LeaderAutoDown at its defaults (unreachable-after 15 s, shut-out restart ON).
Scope:
- Remote exec (WinRM):
- a preflight of service state and recovery config;
- finding the
ScadaBridge-Centralprocess ID through the service (never by image name, becauseapp\is shared withScadaBridge-Site); Stop-Process -Forceon that process ID only;- a post-kill service-state read.
- Logs: pulled over SMB.
- Health: sampled over HTTP from the Mac.
- Not touched: the
ScadaBridge-Siteservices (process IDs unchanged throughout: vq03 7008, vq04 14332), DEV, and every other app.
Preflight (22:43Z)
| Check | Result |
|---|---|
| Central health | vq03 ready 200 / active 503, vq04 ready 200 / active 200. Exactly one active: vq04 |
| Services | ScadaBridge-Central Running (vq03 process 19652, vq04 15040); ScadaBridge-Site Running (vq03 7008, vq04 14332) |
| Recovery | Both centrals restart after 5 s / 5 s / 10 s, reset after 1 day; FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE |
| CLI | instance list --url http://wonder-app-vq04…:8085 returns 11 instances (QA set) |
vq03's Central process had started at 2026-11-15T13:55:26Z, so the active role moved to vq04 before this drill, not during it.
Timings
Times are UTC, measured from the Stop-Process timestamp. Takeover is the first /health/active 200 on the survivor, sampled about every 1.3 s from the Mac (largest gap between samples 2.7 s in Drill A, 3.3 s in Drill B). The other events come from the two Central logs.
| Event | Drill A: kill vq04 (active) | Drill B: kill vq03 (new active) |
|---|---|---|
| Kill | 22:45:19.305, process 15040 | 22:49:24.222, process 19652 |
| SCM starts a new process | 22:45:24.517 (+5.2 s), process 13260 | 22:49:29.347 (+5.1 s), process 27600 |
| Victim cluster node starting | 22:45:28.669 (+9.4 s) | 22:49:32.931 (+8.7 s) |
Survivor: "New incarnation of existing member … is trying to join"; old member marked Down |
22:45:29.222 (+9.9 s) | 22:49:33.223 (+9.0 s) |
Survivor /health/active 200 (takeover) |
22:45:30.525 (+11.2 s); previous sample 22:45:28.992 was 503 | 22:49:33.710 (+9.5 s) |
Victim JOINING → Up (rejoins as standby) |
22:45:39.802 → 22:45:40.299 (+21.0 s) | 22:49:44.010 → 22:49:44.804 (+20.6 s) |
Victim /health/ready 200, /health/active 503 |
22:45:40.091 (+20.8 s) | 22:49:45.069 (+20.8 s) |
| Unavailable window (no active central) | ~11 s | ~9.5 s |
Correction (same day): an earlier version of this record reported a 2-minute sampler gap in Drill A. There was none. The transition summary keyed on the wrong fields and hid vq04's columns. Recomputed from the raw samples, vq04 was ready 200 / active 503 at 22:45:40.091. Drill B had one sample (22:49:35.020) where the survivor vq04's /health/ready returned no response (000) under the sampler's 1 s connect timeout, while its /health/active was 200. The next sample was 200 again.
Finding: the restart beat the downing timer, so [leader-auto-down] … DOWNING did not fire
The pack expected the survivor to log [leader-auto-down] … DOWNING at about +27 s, followed by a takeover at about +30 s. That is the path when the victim stays away. Here the SCM restarted the killed process after 5 s. Its new incarnation reached the survivor at about +9–10 s, well inside unreachable-after (15 s). Akka core then downed the old incarnation itself ("Existing will be removed from the cluster and then new member will be allowed to join"), and leader-auto-down correctly stood aside:
22:45:29.222 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.3:8081, 552687240)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
22:45:29.227 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:45:29.230 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable but already "Down"; not scheduling a Down
22:45:29.305 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:45:40.299 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.3:8081] to ["Up"]
Drill B (survivor vq04):
22:49:33.223 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.2:8081, 1778435843)] is trying to join. …
22:49:33.228 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.2:8081] as ["Down"]
22:49:33.231 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.2:8081 is unreachable but already "Down"; not scheduling a Down
22:49:33.232 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 gained leadership (leader is now akka.tcp://scadabridge@10.220.158.3:8081); pending unreachable=[]
22:49:44.804 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.2:8081] to ["Up"]
22:49:44.805 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 lost leadership (leader is now akka.tcp://scadabridge@10.220.158.2:8081); pending unreachable=[]
On restart, each victim logged [leader-auto-down] started … unreachable-after="00:00:15" and shut-out restart is ON (G6, again).
What this drill proves: with the QA service recovery config, a crashed or killed central is replaced in about 10 s. There's exactly one active central throughout, and the victim rejoins as standby with no operator action. That is faster than the timer path.
What it does not prove: the leader-auto-down timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). The kill drill could not exercise it. The stop-and-hold drill below does.
The Akka leader (the lowest address) and the active node (the oldest Up member) differ after a restart, as documented: after Drill B the leader is vq03 and the active node is vq04.
Final gates (22:51Z)
| Gate | Result |
|---|---|
| One active per pair | Central: vq04 active 200, vq03 503. Site (:8084): vq04 active 200, vq03 503 |
| Ready ×4 | Central vq03/vq04 200; Site vq03/vq04 200 |
| Cluster | Central and site akka-cluster Healthy, memberCount 2, unreachable 0 |
| Site LocalDb | localdb Healthy, replicationConnected: true on both site nodes |
| Sites untouched | ScadaBridge-Site process IDs vq03 7008, vq04 14332, the same as the preflight |
| Inbound API | A bogus X-API-Key on POST /api/SimpleAlarmStatusRequest returns 401 on both nodes |
| CLI | instance list against the new active returned 11 instances after each drill |
Overview (vq03:5320) |
ScadaBridge Up, 4 instances, 2 clusters |
End state: vq04 is the active central, vq03 the standby (Central process 27600). vq04's Central runs process 13260, restarted in Drill A. No config was changed.
Still pending for #116
- QA site-pair drill: pending a Z28061 slot, to go with #142, because it re-hosts every instance.
QA drill of the: done the same day (stop-and-hold drill below).leader-auto-downtimer path- DEV vd03 and its sites; flipping the shipped default; docs and register row 70 (unchanged).
Rollback
None needed: no configuration or binary changed. If a node had not recovered, the plan was Start-Service ScadaBridge-Central on the victim. If both nodes had been down-ed, the plan was to restart both centrals together (SBR DownIfAlone self-downs on a cold recreate).
Stop-and-hold drill (leader-auto-down timer path), 22:55–23:01Z
Owner go: 2026-11-15, "yeah", one leg only, on the current active central. The goal is to keep the victim unreachable: no fast restart, and no graceful leave. A Stop-Service would leave the cluster cleanly (#140), so it would not exercise downing either.
Method: on the active node, vq04:
- Turn off the SCM auto-restart temporarily, using
sc.exe failure ScadaBridge-Central reset= 86400 actions= none/0/none/0/none/0. The script refused to kill if anyRESTARTaction remained. Stop-Process -Forceon the service's process ID.- Hold the victim down for 94 s.
- Restore the recovery config and verify it.
Start-Service.
Recovery config of the victim (vq04 ScadaBridge-Central):
| Before (22:55Z) | During the hold | After the restore (verified before Start-Service) |
|
|---|---|---|---|
RESET_PERIOD |
86400 | 86400 | 86400 |
FAILURE_ACTIONS |
RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms | (none) | RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms |
FAILURE_ACTIONS_ON_NONCRASH_FAILURES |
TRUE | TRUE | TRUE |
The qfailure + qfailureflag output after the restore is byte-identical to the pre-drill output (diff, with the service state lines excluded). A final read at 23:01Z shows the same config on both centrals.
Timings
UTC, relative to the kill.
| Event | Time | After kill |
|---|---|---|
| Kill: vq04 process 13260 (active central) | 22:56:23.574 | 0 |
vq04 /health/* stop answering |
22:56:24.352 | +0.8 s |
| Survivor vq03 marks vq04 UNREACHABLE | 22:56:34.982 | +11.4 s |
[leader-auto-down] schedules the down (15 s) |
22:56:34.985 | +11.4 s |
[leader-auto-down] DOWNING |
22:56:49.990 | +26.4 s |
vq04 marked Down |
22:56:49.991 | +26.4 s |
Survivor vq03 /health/active 200 (takeover) |
22:56:50.314 | +26.7 s |
Leader is removing unreachable node; Member removed; singleton manager Younger → Oldest |
22:56:50.973–.977 | +27.4 s |
Hold ends: config restored, then Start-Service |
22:57:57.298 | +93.7 s |
vq04 cluster node Starting up; Kestrel listening on 8085/8086 |
22:58:01.374 / 22:58:01.768 | start +4.1 s / +4.5 s |
vq03: vq04 JOINING → Up; vq04: Welcome from vq03 |
22:58:01.807 → 22:58:01.980; 22:58:01.852 | start +4.7 s |
vq04 /health/ready 200, /health/active 503 (standby) |
22:58:01.627 (first sample after the listener started) | start +4.3 s |
Unavailable window (no active central): about 26 s. This is the timer path's cost: about 11 s of failure detection plus unreachable-after 15 s. Compare about 10 s when the SCM restarts a killed node.
Downing log, survivor vq03:
22:56:34.982 [WRN] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://scadabridge@10.220.158.3:8081, Uid=552687240 status = Up, role=[Central], upNumber=4, version=1.0.0)]. …
22:56:34.985 [WRN] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable; it will be downed if still unreachable after "00:00:15" (this node akka.tcp://scadabridge@10.220.158.2:8081 is leader: true)
22:56:49.990 [WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8081: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8081) is leader
22:56:49.991 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:56:50.973 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:56:50.976 [INF] Younger observed OldestChanged: [akka.tcp://scadabridge@10.220.158.3:8081 -> myself]
22:56:50.977 [INF] ClusterSingletonManager state change ["Younger" -> "Oldest"] Akka.Cluster.Tools.Singleton.YoungerData
Rejoin: vq04 rejoined as a new incarnation (Welcome from vq03), with no shut-out. The process ID stayed 10160 from Start-Service to the end, so there was no exit-75 self-restart. The downed incarnation had already been removed when the new one started, so the #132 shut-out path did not apply. While its singleton proxies resolved, vq04 logged the expected ClusterSingletonProxy failed to find … after 30 seconds warnings once on start-up; the singletons were identified on vq03 within about 0.1 s.
Final gates (23:01Z)
| Gate | Result |
|---|---|
| One active per pair | Central: vq03 active 200, vq04 503. Site (:8084): vq04 active 200, vq03 503 (site pair untouched) |
| Ready ×4 | Central vq03/vq04 200; Site vq03/vq04 200 |
| Cluster | Central memberCount 2, unreachable 0 on both; site memberCount 2 |
| Site LocalDb | localdb Healthy, replicationConnected: true on both site nodes |
| Sites untouched | ScadaBridge-Site process IDs vq03 7008, vq04 14332 (unchanged all day) |
| Inbound API | A bogus X-API-Key returns 401 on both nodes |
| CLI | instance list against vq03 returns 11 instances |
Overview (vq03:5320) |
ScadaBridge Up, 4 instances, 2 clusters |
| Recovery config | Both centrals: RESTART 5 s / 5 s / 10 s, reset 86400, non-crash TRUE |
End state after all three runs:
- Central: vq03 active (process 27600), vq04 standby (process 10160).
- Site: vq04 active. The site pair was never touched.
Result: the leader-auto-down timer path is proven on QA. An unreachable active central is downed by the leader 15 s after it is detected unreachable, and the survivor takes over with no operator action.