From 1213a743538c446cbb7ce87ad23030eaa1b281fc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Sun, 15 Nov 2026 19:46:47 -0500 Subject: [PATCH] deploy(#116): DEV vd03 central + site on leader-auto-down (config only) One key per file: ScadaBridge:Cluster:SplitBrainResolverStrategy auto-down -> leader-auto-down in appsettings.Central.json and appsettings.Site.json, build 061432c8 unchanged. G6 boot lines on both services, health 200, MxGateway and Z28062 at baseline, bogus key 401, Overview 6/6 Up. Downtime ~23 s central, ~30 s site. No drill: single-node clusters. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8 --- ...-11-15-wonder-app-vd03-leader-auto-down.md | 70 +++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-leader-auto-down.md diff --git a/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-leader-auto-down.md b/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-leader-auto-down.md new file mode 100644 index 00000000..c7a72eb4 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-leader-auto-down.md @@ -0,0 +1,70 @@ +# 2026-11-15 — DEV (wonder-app-vd03) on `leader-auto-down` (#116) + +- **In:** ScadaBridge#116 step "DEV vd03 and its sites". `ScadaBridge:Cluster:SplitBrainResolverStrategy` `auto-down` → `leader-auto-down` in `appsettings.Central.json` and `appsettings.Site.json`, then restarting both services. +- **Out:** + - **Binaries:** unchanged. `061432c8` stays, deployed earlier today, record `2026-11-15-wonder-app-vd03-061432c8.md`. + - **Env and the DB:** unchanged. + - **The shipped default:** unchanged. The default flip is a separate #116 step. + - **No drill:** DEV runs one central and one site, each a single-node cluster. There is no peer to down or take over, so a kill or stop-and-hold drill proves nothing here. The QA drills covered that on 2026-11-15: records `2026-11-15-qa-116-central-drill.md` and `2026-11-15-wonder-app-vq03-vq04-061432c8.md`. +- **Transport:** one `winrm ps -f flip.ps1` call, under owner-authorized deploy scope. It ran the preflight, backup, stop, swap and start. The edited files were staged over SMB, and the gates were checked over HTTP, SMB (logs) and the CLI from the Mac. + +## Preflight (2026-11-16 00:43–00:44Z) + +| Check | Result | +|---|---| +| Build | `061432c8` (deployed 19:13 EST today) | +| Health | central `:8085` and site `:8084`: ready/active 200/200 | +| Strategy | `"auto-down"` in both files | +| Env override | none: Central 12 env entries, Site 6, no `SplitBrain*`/`LeaderAutoDown*` entry | +| Recovery | both services: RESTART 5000/5000/10000 ms, reset 86400, `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` | +| appsettings SHA-256 | `appsettings.Central.json` `3382DD83…`, `appsettings.Site.json` `3E7AD9A4…`, the same as today's deploy record | +| Baseline | MxGateway Connected, 810/810 resolved, good 398 / bad 412; Z28062 Good 144/270, 8 alarms | + +## Change + +- **One byte-exact replacement per file:** `"SplitBrainResolverStrategy": "auto-down"` became `"SplitBrainResolverStrategy": "leader-auto-down"`. Encodings were kept: Central has no BOM, Site has a BOM, and both use LF. +- **Key-level digest diff:** every flattened JSON key was compared by the SHA-256 prefix of its value. The only change in each file is `ScadaBridge:Cluster:SplitBrainResolverStrategy`, `77d79bbc1fb2` → `ea78c76eff79`, the same digests as the QA flip on 2026-11-13. +- **Untouched:** `LeaderAutoDown` stays at its defaults, and `appsettings.json` is not touched. +- **New SHA-256:** Central `5FA256EE…`, Site `804B49DB…`. The script checked them before the swap (staged copies) and after it (live copies). +- **The `_comment_Cluster` text** in both files still says "auto-down". It is a comment, so it was left alone, to keep the change to one key. + +## Window (UTC, from the script and a 1 s Mac-side `/health/active` sampler) + +| Step | Time | +|---|---| +| Backup to `E:\ApiInstall\ScadaBridge\_appsettings-keep-lad-20261115T194403\` | 00:44:05.374 | +| `Stop-Service ScadaBridge-Site` started → stopped | 00:44:05.392 → 00:44:15.688 | +| `Stop-Service ScadaBridge-Central` → stopped | → 00:44:25.972 | +| Files swapped, hashes verified | 00:44:25.98 | +| Central started (pid 18976) / Site started (pid 33572) | 00:44:29.323 / 00:44:30.564 | +| Central `/health/active` back to 200 | 00:44:39.0 | +| Site `/health/active` back to 200 | 00:44:35.9 | + +**Downtime:** central about 23 s (00:44:15.7 → 00:44:39.0), site about 30 s (00:44:06.2 → 00:44:35.9). + +## Gates (all pass) + +| Gate | Result | +|---|---| +| G6 boot log, central | `00:44:29.631 [INF] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@localhost:8081 …`; `00:44:29.637 [INF] [leader-auto-down] started on akka.tcp://scadabridge@localhost:8081: leader=none (this node is leader: false), unreachable-after="00:00:15", already unreachable=[]`; `00:44:34.666 … gained leadership` | +| G6 boot log, site | `00:44:30.897 [INF] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@localhost:8082 …`; `00:44:30.905 [INF] [leader-auto-down] started on akka.tcp://scadabridge@localhost:8082: …`; `00:44:35.940 … gained leadership` | +| Old strategy gone | no `auto-down-unreachable-after` / `keep-majority` deprecation line after the restart; the previous one is at 00:13:45Z, from today's deploy start | +| Errors | 0 `[ERR]`/`[FTL]` after 00:44:25Z in either log | +| Health | central and site ready/active 200/200 | +| MxGateway | Connected, 810/810 resolved, good 398 / bad 412 (= baseline), site report 00:46:00Z | +| Z28062 | Good 144/270, 8 alarms (= baseline) | +| Inbound API | `POST /api/SimpleAlarmStatusRequest` with a bogus `X-API-Key` → **401** | +| Overview `:5320` | 6/6 Up, 0 Degraded / Down / Unreachable / Stale (sweep 19:46:06 EST) | +| Recovery | unchanged, re-read in the preflight: TRUE ×2 | + +## Rollback + +1. Copy both files back from `E:\ApiInstall\ScadaBridge\_appsettings-keep-lad-20261115T194403\` into `E:\ApiInstall\ScadaBridge\app\`. Their hashes are `3382DD83…` / `3E7AD9A4…`. +2. Restart `ScadaBridge-Site` and `ScadaBridge-Central`. +3. Expect the `auto-down-unreachable-after … keep-majority` deprecation warning back in the boot log. + +The staged edited copies stay in `E:\psrem-stage\sb-061432c8\leader-auto-down\`. + +## Note + +On the first attempt the script failed in preflight, before any backup or stop: PowerShell resolves `H` to the built-in `Get-History` alias ahead of a function of the same name. The helper functions were renamed and the script re-run. Nothing on the box changed on the first attempt.