Files
ScadaBridge/docs/deployment/deployment-records/2026-11-15-qa-116-central-drill.md
T
Joseph Doherty 997e36a2bc deploy(#116): QA stop-and-hold drill proves the leader-auto-down timer path
vq04 (active central) killed with SCM auto-restart disabled and held down 94 s:
UNREACHABLE +11.4 s, [leader-auto-down] DOWNING +26.4 s, survivor vq03 active
+26.7 s. Recovery config restored byte-identical before Start-Service; vq04 rejoined
as standby in ~4.7 s with no exit-75. Also corrects the earlier Drill A 'sampler gap'
(an awk keying error; vq04 was ready at +20.8 s).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
2026-11-15 18:02:21 -05:00

14 KiB
Raw Blame History

2026-11-15: #116 leader-auto-down G14 central kill drill, QA vq03/vq04

Owner go: 2026-11-15, "yeah" to running the central drill only. WinRM was explicitly authorized for the drill steps ("i authorize winrm"). The site-pair drill was not run: it re-hosts every instance, including Z28061, which another team uses, and no slot has been agreed. It is still pending and goes with #142.

Build: 8e32f353 on both centrals (deployed 2026-11-13, record 2026-11-13-wonder-app-vq03-vq04-8e32f353.md). ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down, with LeaderAutoDown at its defaults (unreachable-after 15 s, shut-out restart ON).

Scope:

  • Remote exec (WinRM):
    • a preflight of service state and recovery config;
    • finding the ScadaBridge-Central process ID through the service (never by image name, because app\ is shared with ScadaBridge-Site);
    • Stop-Process -Force on that process ID only;
    • a post-kill service-state read.
  • Logs: pulled over SMB.
  • Health: sampled over HTTP from the Mac.
  • Not touched: the ScadaBridge-Site services (process IDs unchanged throughout: vq03 7008, vq04 14332), DEV, and every other app.

Preflight (22:43Z)

Check Result
Central health vq03 ready 200 / active 503, vq04 ready 200 / active 200. Exactly one active: vq04
Services ScadaBridge-Central Running (vq03 process 19652, vq04 15040); ScadaBridge-Site Running (vq03 7008, vq04 14332)
Recovery Both centrals restart after 5 s / 5 s / 10 s, reset after 1 day; FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE
CLI instance list --url http://wonder-app-vq04…:8085 returns 11 instances (QA set)

vq03's Central process had started at 2026-11-15T13:55:26Z, so the active role moved to vq04 before this drill, not during it.

Timings

Times are UTC, measured from the Stop-Process timestamp. Takeover is the first /health/active 200 on the survivor, sampled about every 1.3 s from the Mac (largest gap between samples 2.7 s in Drill A, 3.3 s in Drill B). The other events come from the two Central logs.

Event Drill A: kill vq04 (active) Drill B: kill vq03 (new active)
Kill 22:45:19.305, process 15040 22:49:24.222, process 19652
SCM starts a new process 22:45:24.517 (+5.2 s), process 13260 22:49:29.347 (+5.1 s), process 27600
Victim cluster node starting 22:45:28.669 (+9.4 s) 22:49:32.931 (+8.7 s)
Survivor: "New incarnation of existing member … is trying to join"; old member marked Down 22:45:29.222 (+9.9 s) 22:49:33.223 (+9.0 s)
Survivor /health/active 200 (takeover) 22:45:30.525 (+11.2 s); previous sample 22:45:28.992 was 503 22:49:33.710 (+9.5 s)
Victim JOINING → Up (rejoins as standby) 22:45:39.802 → 22:45:40.299 (+21.0 s) 22:49:44.010 → 22:49:44.804 (+20.6 s)
Victim /health/ready 200, /health/active 503 22:45:40.091 (+20.8 s) 22:49:45.069 (+20.8 s)
Unavailable window (no active central) ~11 s ~9.5 s

Correction (same day): an earlier version of this record reported a 2-minute sampler gap in Drill A. There was none. The transition summary keyed on the wrong fields and hid vq04's columns. Recomputed from the raw samples, vq04 was ready 200 / active 503 at 22:45:40.091. Drill B had one sample (22:49:35.020) where the survivor vq04's /health/ready returned no response (000) under the sampler's 1 s connect timeout, while its /health/active was 200. The next sample was 200 again.

Finding: the restart beat the downing timer, so [leader-auto-down] … DOWNING did not fire

The pack expected the survivor to log [leader-auto-down] … DOWNING at about +27 s, followed by a takeover at about +30 s. That is the path when the victim stays away. Here the SCM restarted the killed process after 5 s. Its new incarnation reached the survivor at about +9–10 s, well inside unreachable-after (15 s). Akka core then downed the old incarnation itself ("Existing will be removed from the cluster and then new member will be allowed to join"), and leader-auto-down correctly stood aside:

22:45:29.222 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.3:8081, 552687240)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
22:45:29.227 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:45:29.230 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable but already "Down"; not scheduling a Down
22:45:29.305 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:45:40.299 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.3:8081] to ["Up"]

Drill B (survivor vq04):

22:49:33.223 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.2:8081, 1778435843)] is trying to join. …
22:49:33.228 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.2:8081] as ["Down"]
22:49:33.231 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.2:8081 is unreachable but already "Down"; not scheduling a Down
22:49:33.232 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 gained leadership (leader is now akka.tcp://scadabridge@10.220.158.3:8081); pending unreachable=[]
22:49:44.804 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.2:8081] to ["Up"]
22:49:44.805 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 lost leadership (leader is now akka.tcp://scadabridge@10.220.158.2:8081); pending unreachable=[]

On restart, each victim logged [leader-auto-down] started … unreachable-after="00:00:15" and shut-out restart is ON (G6, again).

What this drill proves: with the QA service recovery config, a crashed or killed central is replaced in about 10 s. There's exactly one active central throughout, and the victim rejoins as standby with no operator action. That is faster than the timer path.

What it does not prove: the leader-auto-down timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). The kill drill could not exercise it. The stop-and-hold drill below does.

The Akka leader (the lowest address) and the active node (the oldest Up member) differ after a restart, as documented: after Drill B the leader is vq03 and the active node is vq04.

Final gates (22:51Z)

Gate Result
One active per pair Central: vq04 active 200, vq03 503. Site (:8084): vq04 active 200, vq03 503
Ready ×4 Central vq03/vq04 200; Site vq03/vq04 200
Cluster Central and site akka-cluster Healthy, memberCount 2, unreachable 0
Site LocalDb localdb Healthy, replicationConnected: true on both site nodes
Sites untouched ScadaBridge-Site process IDs vq03 7008, vq04 14332, the same as the preflight
Inbound API A bogus X-API-Key on POST /api/SimpleAlarmStatusRequest returns 401 on both nodes
CLI instance list against the new active returned 11 instances after each drill
Overview (vq03:5320) ScadaBridge Up, 4 instances, 2 clusters

End state: vq04 is the active central, vq03 the standby (Central process 27600). vq04's Central runs process 13260, restarted in Drill A. No config was changed.

Still pending for #116

  • QA site-pair drill: pending a Z28061 slot, to go with #142, because it re-hosts every instance.
  • QA drill of the leader-auto-down timer path: done the same day (stop-and-hold drill below).
  • DEV vd03 and its sites; flipping the shipped default; docs and register row 70 (unchanged).

Rollback

None needed: no configuration or binary changed. If a node had not recovered, the plan was Start-Service ScadaBridge-Central on the victim. If both nodes had been down-ed, the plan was to restart both centrals together (SBR DownIfAlone self-downs on a cold recreate).

Stop-and-hold drill (leader-auto-down timer path), 22:55–23:01Z

Owner go: 2026-11-15, "yeah", one leg only, on the current active central. The goal is to keep the victim unreachable: no fast restart, and no graceful leave. A Stop-Service would leave the cluster cleanly (#140), so it would not exercise downing either.

Method: on the active node, vq04:

  1. Turn off the SCM auto-restart temporarily, using sc.exe failure ScadaBridge-Central reset= 86400 actions= none/0/none/0/none/0. The script refused to kill if any RESTART action remained.
  2. Stop-Process -Force on the service's process ID.
  3. Hold the victim down for 94 s.
  4. Restore the recovery config and verify it.
  5. Start-Service.

Recovery config of the victim (vq04 ScadaBridge-Central):

Before (22:55Z) During the hold After the restore (verified before Start-Service)
RESET_PERIOD 86400 86400 86400
FAILURE_ACTIONS RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms (none) RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms
FAILURE_ACTIONS_ON_NONCRASH_FAILURES TRUE TRUE TRUE

The qfailure + qfailureflag output after the restore is byte-identical to the pre-drill output (diff, with the service state lines excluded). A final read at 23:01Z shows the same config on both centrals.

Timings

UTC, relative to the kill.

Event Time After kill
Kill: vq04 process 13260 (active central) 22:56:23.574 0
vq04 /health/* stop answering 22:56:24.352 +0.8 s
Survivor vq03 marks vq04 UNREACHABLE 22:56:34.982 +11.4 s
[leader-auto-down] schedules the down (15 s) 22:56:34.985 +11.4 s
[leader-auto-down] DOWNING 22:56:49.990 +26.4 s
vq04 marked Down 22:56:49.991 +26.4 s
Survivor vq03 /health/active 200 (takeover) 22:56:50.314 +26.7 s
Leader is removing unreachable node; Member removed; singleton manager Younger → Oldest 22:56:50.973–.977 +27.4 s
Hold ends: config restored, then Start-Service 22:57:57.298 +93.7 s
vq04 cluster node Starting up; Kestrel listening on 8085/8086 22:58:01.374 / 22:58:01.768 start +4.1 s / +4.5 s
vq03: vq04 JOINING → Up; vq04: Welcome from vq03 22:58:01.807 → 22:58:01.980; 22:58:01.852 start +4.7 s
vq04 /health/ready 200, /health/active 503 (standby) 22:58:01.627 (first sample after the listener started) start +4.3 s

Unavailable window (no active central): about 26 s. This is the timer path's cost: about 11 s of failure detection plus unreachable-after 15 s. Compare about 10 s when the SCM restarts a killed node.

Downing log, survivor vq03:

22:56:34.982 [WRN] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://scadabridge@10.220.158.3:8081, Uid=552687240 status = Up, role=[Central], upNumber=4, version=1.0.0)]. …
22:56:34.985 [WRN] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable; it will be downed if still unreachable after "00:00:15" (this node akka.tcp://scadabridge@10.220.158.2:8081 is leader: true)
22:56:49.990 [WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8081: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8081) is leader
22:56:49.991 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:56:50.973 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:56:50.976 [INF] Younger observed OldestChanged: [akka.tcp://scadabridge@10.220.158.3:8081 -> myself]
22:56:50.977 [INF] ClusterSingletonManager state change ["Younger" -> "Oldest"] Akka.Cluster.Tools.Singleton.YoungerData

Rejoin: vq04 rejoined as a new incarnation (Welcome from vq03), with no shut-out. The process ID stayed 10160 from Start-Service to the end, so there was no exit-75 self-restart. The downed incarnation had already been removed when the new one started, so the #132 shut-out path did not apply. While its singleton proxies resolved, vq04 logged the expected ClusterSingletonProxy failed to find … after 30 seconds warnings once on start-up; the singletons were identified on vq03 within about 0.1 s.

Final gates (23:01Z)

Gate Result
One active per pair Central: vq03 active 200, vq04 503. Site (:8084): vq04 active 200, vq03 503 (site pair untouched)
Ready ×4 Central vq03/vq04 200; Site vq03/vq04 200
Cluster Central memberCount 2, unreachable 0 on both; site memberCount 2
Site LocalDb localdb Healthy, replicationConnected: true on both site nodes
Sites untouched ScadaBridge-Site process IDs vq03 7008, vq04 14332 (unchanged all day)
Inbound API A bogus X-API-Key returns 401 on both nodes
CLI instance list against vq03 returns 11 instances
Overview (vq03:5320) ScadaBridge Up, 4 instances, 2 clusters
Recovery config Both centrals: RESTART 5 s / 5 s / 10 s, reset 86400, non-crash TRUE

End state after all three runs:

  • Central: vq03 active (process 27600), vq04 standby (process 10160).
  • Site: vq04 active. The site pair was never touched.

Result: the leader-auto-down timer path is proven on QA. An unreachable active central is downed by the leader 15 s after it is detected unreachable, and the survivor takes over with no operator action.