deploy(#116): QA stop-and-hold drill proves the leader-auto-down timer path

vq04 (active central) killed with SCM auto-restart disabled and held down 94 s:
UNREACHABLE +11.4 s, [leader-auto-down] DOWNING +26.4 s, survivor vq03 active
+26.7 s. Recovery config restored byte-identical before Start-Service; vq04 rejoined
as standby in ~4.7 s with no exit-75. Also corrects the earlier Drill A 'sampler gap'
(an awk keying error; vq04 was ready at +20.8 s).

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
Joseph Doherty
2026-11-15 18:02:21 -05:00
parent cae5ab958c
commit 997e36a2bc
@@ -27,7 +27,7 @@ vq03's Central process had started at 2026-11-15T13:55:26Z, so the active role m
## Timings
Times are UTC, measured from the `Stop-Process` timestamp. Takeover is the first `/health/active` 200 on the survivor, sampled about every 1.3 s from the Mac. The other events come from the two Central logs.
Times are UTC, measured from the `Stop-Process` timestamp. Takeover is the first `/health/active` 200 on the survivor, sampled about every 1.3 s from the Mac (largest gap between samples 2.7 s in Drill A, 3.3 s in Drill B). The other events come from the two Central logs.
| Event | Drill A: kill vq04 (active) | Drill B: kill vq03 (new active) |
|---|---|---|
@@ -37,10 +37,10 @@ Times are UTC, measured from the `Stop-Process` timestamp. Takeover is the first
| Survivor: "New incarnation of existing member … is trying to join"; old member marked `Down` | 22:45:29.222 (**+9.9 s**) | 22:49:33.223 (**+9.0 s**) |
| Survivor `/health/active` 200 (takeover) | 22:45:30.525 (**+11.2 s**); previous sample 22:45:28.992 was 503 | 22:49:33.710 (**+9.5 s**) |
| Victim `JOINING` → `Up` (rejoins as standby) | 22:45:39.802 → 22:45:40.299 (**+21.0 s**) | 22:49:44.010 → 22:49:44.804 (**+20.6 s**) |
| Victim `/health/ready` 200, `/health/active` 503 | by 22:47:37 (sampler gap, below); `akka-cluster` health errors stop at 22:45:39 | 22:49:45.069 (**+20.8 s**) |
| Victim `/health/ready` 200, `/health/active` 503 | 22:45:40.091 (**+20.8 s**) | 22:49:45.069 (**+20.8 s**) |
| Unavailable window (no active central) | ~11 s | ~9.5 s |
**Sampler gap, Drill A:** the Mac-side sampler stalled from 22:45:37 to 22:47:37. A curl hung past its `-m` limit, and Drill B ran with `--connect-timeout`. The victim's Up time comes from the survivor's log (22:45:40.3), and its health-check errors stop at 22:45:39. It was ready 200 / active 503 by the next sample.
**Correction (same day):** an earlier version of this record reported a 2-minute sampler gap in Drill A. There was none. The transition summary keyed on the wrong fields and hid vq04's columns. Recomputed from the raw samples, vq04 was ready 200 / active 503 at 22:45:40.091. Drill B had one sample (22:49:35.020) where the survivor vq04's `/health/ready` returned no response (`000`) under the sampler's 1 s connect timeout, while its `/health/active` was 200. The next sample was 200 again.
## Finding: the restart beat the downing timer, so `[leader-auto-down] … DOWNING` did not fire
@@ -69,7 +69,7 @@ On restart, each victim logged `[leader-auto-down] started … unreachable-after
**What this drill proves:** with the QA service recovery config, a crashed or killed central is replaced in about 10 s. There's exactly one active central throughout, and the victim rejoins as standby with no operator action. That is faster than the timer path.
**What it does not prove:** the `leader-auto-down` timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). It is proven on the rig and on docker-env2 (#116 checklist), not on QA. To exercise it on QA, the victim must stay away for more than 15 s (`Stop-Service`, with the SCM not restarting it, or a firewall block), and that is a different drill.
**What it does not prove:** the `leader-auto-down` timer path itself, where a node stays unreachable for more than 15 s (for example, a hung process or a network loss rather than a crash). The kill drill could not exercise it. The **stop-and-hold drill** below does.
The Akka leader (the lowest address) and the active node (the oldest `Up` member) differ after a restart, as documented: after Drill B the leader is vq03 and the active node is vq04.
@@ -91,9 +91,85 @@ The Akka leader (the lowest address) and the active node (the oldest `Up` member
## Still pending for #116
- **QA site-pair drill:** pending a Z28061 slot, to go with #142, because it re-hosts every instance.
- Optional: a QA drill of the `leader-auto-down` timer path (victim held away for more than 15 s), if the owner wants that on QA rather than relying on the rig and env2 evidence.
- ~~QA drill of the `leader-auto-down` timer path~~: done the same day (stop-and-hold drill below).
- DEV vd03 and its sites; flipping the shipped default; docs and register row 70 (unchanged).
## Rollback
None needed: no configuration or binary changed. If a node had not recovered, the plan was `Start-Service ScadaBridge-Central` on the victim. If both nodes had been down-ed, the plan was to restart both centrals together (SBR `DownIfAlone` self-downs on a cold recreate).
## Stop-and-hold drill (leader-auto-down timer path), 22:55–23:01Z
**Owner go:** 2026-11-15, "yeah", one leg only, on the current active central. The goal is to keep the victim **unreachable**: no fast restart, and no graceful leave. A `Stop-Service` would leave the cluster cleanly (#140), so it would not exercise downing either.
**Method:** on the active node, vq04:
1. Turn off the SCM auto-restart temporarily, using `sc.exe failure ScadaBridge-Central reset= 86400 actions= none/0/none/0/none/0`. The script refused to kill if any `RESTART` action remained.
2. `Stop-Process -Force` on the service's process ID.
3. Hold the victim down for 94 s.
4. Restore the recovery config and verify it.
5. `Start-Service`.
**Recovery config of the victim (vq04 `ScadaBridge-Central`):**
| | Before (22:55Z) | During the hold | After the restore (verified before `Start-Service`) |
|---|---|---|---|
| `RESET_PERIOD` | 86400 | 86400 | 86400 |
| `FAILURE_ACTIONS` | RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms | *(none)* | RESTART 5000 ms / RESTART 5000 ms / RESTART 10000 ms |
| `FAILURE_ACTIONS_ON_NONCRASH_FAILURES` | TRUE | TRUE | TRUE |
The `qfailure` + `qfailureflag` output after the restore is **byte-identical** to the pre-drill output (`diff`, with the service state lines excluded). A final read at 23:01Z shows the same config on both centrals.
### Timings
UTC, relative to the kill.
| Event | Time | After kill |
|---|---|---|
| Kill: vq04 process 13260 (active central) | 22:56:23.574 | 0 |
| vq04 `/health/*` stop answering | 22:56:24.352 | +0.8 s |
| Survivor vq03 marks vq04 **UNREACHABLE** | 22:56:34.982 | **+11.4 s** |
| `[leader-auto-down]` schedules the down (15 s) | 22:56:34.985 | +11.4 s |
| `[leader-auto-down] DOWNING` | 22:56:49.990 | **+26.4 s** |
| vq04 marked `Down` | 22:56:49.991 | +26.4 s |
| Survivor vq03 `/health/active` 200 (takeover) | 22:56:50.314 | **+26.7 s** |
| `Leader is removing unreachable node`; `Member removed`; singleton manager `Younger → Oldest` | 22:56:50.973–.977 | +27.4 s |
| Hold ends: config restored, then `Start-Service` | 22:57:57.298 | +93.7 s |
| vq04 cluster node `Starting up`; Kestrel listening on 8085/8086 | 22:58:01.374 / 22:58:01.768 | start +4.1 s / +4.5 s |
| vq03: vq04 **JOINING** → **Up**; vq04: `Welcome from` vq03 | 22:58:01.807 → 22:58:01.980; 22:58:01.852 | start +4.7 s |
| vq04 `/health/ready` 200, `/health/active` 503 (standby) | 22:58:01.627 (first sample after the listener started) | start +4.3 s |
**Unavailable window (no active central): about 26 s.** This is the timer path's cost: about 11 s of failure detection plus `unreachable-after` 15 s. Compare about 10 s when the SCM restarts a killed node.
**Downing log, survivor vq03:**
```
22:56:34.982 [WRN] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking node(s) as UNREACHABLE [Member(address = akka.tcp://scadabridge@10.220.158.3:8081, Uid=552687240 status = Up, role=[Central], upNumber=4, version=1.0.0)]. …
22:56:34.985 [WRN] [leader-auto-down] akka.tcp://scadabridge@10.220.158.3:8081 is unreachable; it will be downed if still unreachable after "00:00:15" (this node akka.tcp://scadabridge@10.220.158.2:8081 is leader: true)
22:56:49.990 [WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8081: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8081) is leader
22:56:49.991 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Marking unreachable node [akka.tcp://scadabridge@10.220.158.3:8081] as ["Down"]
22:56:50.973 [INF] Cluster Node [akka.tcp://scadabridge@10.220.158.2:8081] - Leader is removing unreachable node [akka.tcp://scadabridge@10.220.158.3:8081]
22:56:50.976 [INF] Younger observed OldestChanged: [akka.tcp://scadabridge@10.220.158.3:8081 -> myself]
22:56:50.977 [INF] ClusterSingletonManager state change ["Younger" -> "Oldest"] Akka.Cluster.Tools.Singleton.YoungerData
```
**Rejoin:** vq04 rejoined as a new incarnation (`Welcome from` vq03), with no shut-out. The process ID stayed 10160 from `Start-Service` to the end, so there was **no exit-75 self-restart**. The downed incarnation had already been removed when the new one started, so the #132 shut-out path did not apply. While its singleton proxies resolved, vq04 logged the expected `ClusterSingletonProxy failed to find … after 30 seconds` warnings once on start-up; the singletons were identified on vq03 within about 0.1 s.
### Final gates (23:01Z)
| Gate | Result |
|---|---|
| One active per pair | Central: **vq03** active 200, vq04 503. Site (`:8084`): vq04 active 200, vq03 503 (site pair untouched) |
| Ready ×4 | Central vq03/vq04 200; Site vq03/vq04 200 |
| Cluster | Central memberCount 2, unreachable 0 on both; site memberCount 2 |
| Site LocalDb | `localdb` Healthy, `replicationConnected: true` on both site nodes |
| Sites untouched | `ScadaBridge-Site` process IDs vq03 7008, vq04 14332 (unchanged all day) |
| Inbound API | A bogus `X-API-Key` returns **401** on both nodes |
| CLI | `instance list` against vq03 returns 11 instances |
| Overview (`vq03:5320`) | ScadaBridge Up, 4 instances, 2 clusters |
| Recovery config | Both centrals: RESTART 5 s / 5 s / 10 s, reset 86400, non-crash TRUE |
**End state after all three runs:**
- **Central:** vq03 active (process 27600), vq04 standby (process 10160).
- **Site:** vq04 active. The site pair was never touched.
**Result:** the `leader-auto-down` timer path is **proven on QA**. An unreachable active central is downed by the leader 15 s after it is detected unreachable, and the survivor takes over with no operator action.