deploy(#116): QA central kill drill (G14 central) — takeover ~10 s via new-incarnation path
Both centrals killed in turn (Stop-Process on the service PID). SCM restart in ~5 s; the survivor downed the old incarnation at ~+9-10 s and took the active role; the victim rejoined as standby at ~+21 s. leader-auto-down stood aside (already Down), so its 15 s timer path was not exercised on QA. Site-pair drill still pending (Z28061 slot, #142). Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -0,0 +1,99 @@
|
|||||||
|
# 2026-11-15: #116 `leader-auto-down` G14 central kill drill, QA vq03/vq04
|
||||||
|
|
||||||
|
**Owner go:** 2026-11-15, "yeah" to running the **central** drill only. WinRM was explicitly authorized for the drill steps ("i authorize winrm"). The **site-pair drill was not run**: it re-hosts every instance, including Z28061, which another team uses, and no slot has been agreed. It is still pending and goes with #142.
|
||||||
|
|
||||||
|
**Build:** `8e32f353` on both centrals (deployed 2026-11-13, record `2026-11-13-wonder-app-vq03-vq04-8e32f353.md`). `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down`, with `LeaderAutoDown` at its defaults (`unreachable-after` 15 s, shut-out restart ON).
|
||||||
|
|
||||||
|
**Scope:**
|
||||||
|
- **Remote exec (WinRM):**
|
||||||
|
- a preflight of service state and recovery config;
|
||||||
|
- finding the `ScadaBridge-Central` process ID through the service (never by image name, because `app\` is shared with `ScadaBridge-Site`);
|
||||||
|
- `Stop-Process -Force` on that process ID only;
|
||||||
|
- a post-kill service-state read.
|
||||||
|
- **Logs:** pulled over SMB.
|
||||||
|
- **Health:** sampled over HTTP from the Mac.
|
||||||
|
- **Not touched:** the `ScadaBridge-Site` services (process IDs unchanged throughout: vq03 7008, vq04 14332), DEV, and every other app.
|
||||||
|
|
||||||
|
## Preflight (22:43Z)
|
||||||
|
|
||||||
|
| Check | Result |
|
||||||
|
|---|---|
|
||||||
|
| Central health | vq03 ready 200 / active 503, vq04 ready 200 / active **200**. Exactly one active: **vq04** |
|
||||||
|
| Services | `ScadaBridge-Central` Running (vq03 process 19652, vq04 15040); `ScadaBridge-Site` Running (vq03 7008, vq04 14332) |
|
||||||
|
| Recovery | Both centrals restart after 5 s / 5 s / 10 s, reset after 1 day; `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` |
|
||||||
|
| CLI | `instance list --url http://wonder-app-vq04…:8085` returns 11 instances (QA set) |
|
||||||
|
|
||||||
|
vq03's Central process had started at 2026-11-15T13:55:26Z, so the active role moved to vq04 before this drill, not during it.
|
||||||
|
|
||||||
|
## Timings
|
||||||
|
|
||||||
|
Times are UTC, measured from the `Stop-Process` timestamp. Takeover is the first `/health/active` 200 on the survivor, sampled about every 1.3 s from the Mac. The other events come from the two Central logs.
|
||||||
|
|
||||||
|
| Event | Drill A: kill vq04 (active) | Drill B: kill vq03 (new active) |
|
||||||
|
|---|---|---|
|
||||||
|
| Kill | 22:45:19.305, process 15040 | 22:49:24.222, process 19652 |
|
||||||
|
| SCM starts a new process | 22:45:24.517 (**+5.2 s**), process 13260 | 22:49:29.347 (**+5.1 s**), process 27600 |
|
||||||
|
| Victim cluster node starting | 22:45:28.669 (+9.4 s) | 22:49:32.931 (+8.7 s) |
|
||||||
|
| Survivor: "New incarnation of existing member … is trying to join"; old member marked `Down` | 22:45:29.222 (**+9.9 s**) | 22:49:33.223 (**+9.0 s**) |
|
||||||
|
| Survivor `/health/active` 200 (takeover) | 22:45:30.525 (**+11.2 s**); previous sample 22:45:28.992 was 503 | 22:49:33.710 (**+9.5 s**) |
|
||||||
|
| Victim `JOINING` → `Up` (rejoins as standby) | 22:45:39.802 → 22:45:40.299 (**+21.0 s**) | 22:49:44.010 → 22:49:44.804 (**+20.6 s**) |
|
||||||
|
| Victim `/health/ready` 200, `/health/active` 503 | by 22:47:37 (sampler gap, below); `akka-cluster` health errors stop at 22:45:39 | 22:49:45.069 (**+20.8 s**) |
|
||||||
|
| Unavailable window (no active central) | ~11 s | ~9.5 s |
|
||||||
|
|
||||||
|
**Sampler gap, Drill A:** the Mac-side sampler stalled from 22:45:37 to 22:47:37. A curl hung past its `-m` limit, and Drill B ran with `--connect-timeout`. The victim's Up time comes from the survivor's log (22:45:40.3), and its health-check errors stop at 22:45:39. It was ready 200 / active 503 by the next sample.
|
||||||
|
|
||||||
|
## Finding: the restart beat the downing timer, so `[leader-auto-down] … DOWNING` did not fire
|
||||||
|
|
||||||
|
The pack expected the survivor to log `[leader-auto-down] … DOWNING` at about +27 s, followed by a takeover at about +30 s. That is the path when the victim **stays** away. Here the SCM restarted the killed process after 5 s. Its new incarnation reached the survivor at about +9–10 s, well inside `unreachable-after` (15 s). Akka core then downed the old incarnation itself ("Existing will be removed from the cluster and then new member will be allowed to join"), and `leader-auto-down` correctly stood aside:
|
||||||
|
|
||||||
|
```
|
||||||
|
22:45:29.222 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.3:8081, 552687240)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
|
||||||
|
22:45:29.227 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
|
||||||
|
22:45:29.230 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable but already "Down"; not scheduling a Down
|
||||||
|
22:45:29.305 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
|
||||||
|
22:45:40.299 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.3:8081] to ["Up"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Drill B (survivor vq04):
|
||||||
|
|
||||||
|
```
|
||||||
|
22:49:33.223 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@10.220.158.2:8081, 1778435843)] is trying to join. …
|
||||||
|
22:49:33.228 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.2:8081] as ["Down"]
|
||||||
|
22:49:33.231 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.2:8081 is unreachable but already "Down"; not scheduling a Down
|
||||||
|
22:49:33.232 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 gained leadership (leader is now akka.tcp://scadabridge@10.220.158.3:8081); pending unreachable=[]
|
||||||
|
22:49:44.804 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.3:8081] - Leader is moving node [akka.tcp://scadabridge@10.220.158.2:8081] to ["Up"]
|
||||||
|
22:49:44.805 [INF] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 lost leadership (leader is now akka.tcp://scadabridge@10.220.158.2:8081); pending unreachable=[]
|
||||||
|
```
|
||||||
|
|
||||||
|
On restart, each victim logged `[leader-auto-down] started … unreachable-after="00:00:15"` and `shut-out restart is ON` (G6, again).
|
||||||
|
|
||||||
|
**What this drill proves:** with the QA service recovery config, a crashed or killed central is replaced in about 10 s. There's exactly one active central throughout, and the victim rejoins as standby with no operator action. That is faster than the timer path.
|
||||||
|
|
||||||
|
**What it does not prove:** the `leader-auto-down` timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). It is proven on the rig and on docker-env2 (#116 checklist), not on QA. To exercise it on QA, the victim must stay away for more than 15 s (`Stop-Service`, with the SCM not restarting it, or a firewall block), and that is a different drill.
|
||||||
|
|
||||||
|
The Akka leader (the lowest address) and the active node (the oldest `Up` member) differ after a restart, as documented: after Drill B the leader is vq03 and the active node is vq04.
|
||||||
|
|
||||||
|
## Final gates (22:51Z)
|
||||||
|
|
||||||
|
| Gate | Result |
|
||||||
|
|---|---|
|
||||||
|
| One active per pair | Central: vq04 active 200, vq03 503. Site (`:8084`): vq04 active 200, vq03 503 |
|
||||||
|
| Ready ×4 | Central vq03/vq04 200; Site vq03/vq04 200 |
|
||||||
|
| Cluster | Central and site `akka-cluster` Healthy, memberCount 2, unreachable 0 |
|
||||||
|
| Site LocalDb | `localdb` Healthy, `replicationConnected: true` on both site nodes |
|
||||||
|
| Sites untouched | `ScadaBridge-Site` process IDs vq03 7008, vq04 14332, the same as the preflight |
|
||||||
|
| Inbound API | A bogus `X-API-Key` on `POST /api/SimpleAlarmStatusRequest` returns **401** on both nodes |
|
||||||
|
| CLI | `instance list` against the new active returned 11 instances after each drill |
|
||||||
|
| Overview (`vq03:5320`) | ScadaBridge **Up**, 4 instances, 2 clusters |
|
||||||
|
|
||||||
|
**End state:** vq04 is the active central, vq03 the standby (Central process 27600). vq04's Central runs process 13260, restarted in Drill A. No config was changed.
|
||||||
|
|
||||||
|
## Still pending for #116
|
||||||
|
|
||||||
|
- **QA site-pair drill:** pending a Z28061 slot, to go with #142, because it re-hosts every instance.
|
||||||
|
- Optional: a QA drill of the `leader-auto-down` timer path (victim held away for more than 15 s), if the owner wants that on QA rather than relying on the rig and env2 evidence.
|
||||||
|
- DEV vd03 and its sites; flipping the shipped default; docs and register row 70 (unchanged).
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
None needed: no configuration or binary changed. If a node had not recovered, the plan was `Start-Service ScadaBridge-Central` on the victim. If both nodes had been down-ed, the plan was to restart both centrals together (SBR `DownIfAlone` self-downs on a cold recreate).
|
||||||
Reference in New Issue
Block a user