Joseph Doherty
17864d1ae4
Both driven from the Mac over the winrm CLI (WinRM 5985 + SMB 445, no helper on the boxes). Migration 53->55; the first Central boot encrypts DataConnections.PrimaryConfiguration (dp:protocol:v1:), so rollback is the COPY_ONLY DB restore, not a binary swap. QA: all four nodes stopped together, one shared-DB migration, 2 min 35 s window, preflight against a restored clone. Claude-Session: https://claude.ai/code/session_01RgthVLVU5CJ8SUuEwruTkt
103 lines
7.3 KiB
Markdown
103 lines
7.3 KiB
Markdown
# 2026-09-10 — DEV (wonder-app-vd03) → `2b9a7c3c` (arch-review remediation + CLI parity + Flow Debugger v1.4)
|
|
|
|
**Build:** `main` @ `2b9a7c3c` (= `origin/main`), ProductVersion `1.0.0+2b9a7c3ca05fde8fe58f647d8f91e5ab9452807b`,
|
|
published from a detached worktree at that SHA (`dotnet publish` Host + CLI `-c Release -r win-x64
|
|
--self-contained true -p:PublishSingleFile=false`; 1029 app files, 223 cli files, zip 125 543 056 B,
|
|
SHA-256 `371b3271…`).
|
|
**Delta from the box's previous build `afb27e77`:** 253 commits — Flow Debugger instance picker + v1.4
|
|
step logs, the 2026-09-09 whole-system arch review remediation (13 High + 33 Medium + 46 Low), and
|
|
management API / CLI parity (`docs/plans/2026-09-10-management-api-cli-parity-review.md`). The CLI
|
|
gains the `shared-schema` and `flow-debug` groups.
|
|
**QA (vq03/vq04) NOT touched** — this record covers vd03 only.
|
|
|
|
**Transport:** first ScadaBridge deploy driven entirely by the Mac-side `winrm` CLI (WinRM 5985/NTLM
|
|
for commands, `winrm smb push` over 445 for the zip) — no psbridge, no servecli.
|
|
|
|
## Schema: 53 → 55 migrations, one of them a coordinated data upgrade
|
|
|
|
| Migration | Effect | Pre-check |
|
|
|---|---|---|
|
|
| `20260906034113_AddApiMethodTraceAllCalls` | `ApiMethods.TraceAllCalls bit NOT NULL DEFAULT 0` | — |
|
|
| `20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` | `THROW 51002` if `Instances.UniqueName` is not globally unique; drops `IX_Instances_SiteId_UniqueName`, adds `IX_Instances_SiteId` + unique `IX_Instances_UniqueName`; widens `DataConnections.Primary/BackupConfiguration` to `nvarchar(max)` | 0 duplicate names (4 instances, all site 2); 1 DataConnection (`MxGateway`, 210 chars, unprotected) |
|
|
|
|
The migration is schema-only. **The first Central boot on the new binary ran
|
|
`DataConnectionSecretUpgrade`** and rewrote `DataConnections.PrimaryConfiguration` as
|
|
`dp:protocol:v1:…` (LEN 210 → 426) using the Data Protection key ring, which on this install lives
|
|
**in the same database** (`DataProtectionKeys`, 2 keys, unchanged). So the single
|
|
`BACKUP DATABASE … COPY_ONLY, CHECKSUM` taken beforehand *is* the coordinated database/key-ring
|
|
backup the component doc asks for. **Old binaries cannot read the protected row, and the migration's
|
|
`Down` refuses while protected rows exist** — rollback is the DB restore, not `dotnet ef`.
|
|
|
|
From→to script generated with `dotnet ef migrations script 20260819201210_AddNotificationsListSortIndex
|
|
20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration` (NOT idempotent — the
|
|
full idempotent script does not compile against this DB, see the 2026-06-16 record), applied with
|
|
`sqlcmd -C -b -I` while both services were stopped.
|
|
|
|
## Config
|
|
|
|
**No appsettings change.** Every new option in the delta has a passing default
|
|
(`InboundApi:StandbyForwardPeerEndpoints`/`StandbyForwardTimeout`/`MaxLogLinesPerRun`,
|
|
`SiteRuntime:TagSubscribeRetryMaxIntervalMs`/`TagSubscribeInFlightCeilingMs`/`TraceArmWindowSeconds`/
|
|
`MaxLogLinesPerRun`, `Communication:ConnectedHeaderTimeout`, `MxGateway:ClientNamePrefix`,
|
|
`AuditLogPurge:MaxPurgedRowsPerChannelPerTick`), verified by reading the validators and then by the
|
|
clean boot. `ScadaBridge:InboundApi:ApiKeyStore:Provider` **left unset (= Sqlite)** — the SqlServer
|
|
cutover is the separate planned record `2026-09-XX-zb-servers-inbound-key-store.md`. The box's three
|
|
`appsettings*.json` were copied out, restored after the swap and SHA-256-matched. Service registry
|
|
environment untouched.
|
|
|
|
## Procedure (`E:\psrem-stage\2b9a7c3c\sb-swap.ps1`, log `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-20260910T104529.txt`)
|
|
|
|
1. 10:43 — `reg export` of both service keys (`_envbak-ScadaBridge-{Central,Site}-20260910T104349.reg`);
|
|
`BACKUP DATABASE [ScadaBridge] … COPY_ONLY, CHECKSUM, COMPRESSION` →
|
|
`E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak` (120 MB),
|
|
`RESTORE VERIFYONLY … WITH CHECKSUM` ok. Zip pushed over SMB, SHA-256 verified on the box, unpacked,
|
|
staged `Host.exe`/`scadabridge.exe` ProductVersion checked = `2b9a7c3c`.
|
|
2. Baseline: `/health/ready` 200/200; `POST /management ListInstances` no-auth → 200; bogus
|
|
`X-API-Key` → 401; `debug snapshot` Z28061Sim(8) attrs=270 Bad=28 Good=242, Z28061(1) attrs=270
|
|
Bad=126 Good=144 (via the on-box CLI with `--username multi-role --password password`).
|
|
3. **10:45:29** Site stopped (10 s), Central stopped (10 s), no `Host.exe` left.
|
|
4. 10:45:50 migration script: 53 → 55 rows in `__EFMigrationsHistory`.
|
|
5. `app`/`cli` → `app.bak.20260910T104529` / `cli.bak.20260910T104529`; staged copy in; the three
|
|
appsettings restored (SHA match).
|
|
6. 10:45:57 Central started → ready 10:46:11 (14 s, includes the protocol-row encryption upgrade);
|
|
Site started → ready 10:46:19. **~50 s total downtime.**
|
|
|
|
## Verification
|
|
|
|
| Check | Result |
|
|
|---|---|
|
|
| `:8085 /health/ready` | 200 Healthy — database, akka-cluster Up (members=1), required-singletons 5/5 |
|
|
| `:8084 /health/ready` | 200 Healthy — akka-cluster, localdb |
|
|
| ProductVersion | Host + CLI + 25 `ZB.MOM.WW.ScadaBridge.*` DLLs at `2b9a7c3c` |
|
|
| DB | 55 migrations; new indexes present; `DataConnections` id 1 `PrimaryConfiguration` protected (`dp:protocol:v1:`), backup null; `DataProtectionKeys` = 2 |
|
|
| Auth posture | `POST /management` no-auth 200 (DisableLogin preserved); bogus `X-API-Key` 401 (key store loaded); `GET /` 200 |
|
|
| Site runtime | Loaded 3 deployed configs; reconcile 0 fetched / 0 failed / 0 orphans; Z28061, Z28061Sim, Z28062 each `subscribing to 270 tags on connection MxGateway` |
|
|
| Snapshots after | Z28061Sim attrs=270 alarms=1 Bad=28 Good=242; Z28061 attrs=270 Bad=126 Good=144 — **identical to baseline** |
|
|
| Logs after boot | central 0 ERR/FTL, site 0 ERR/FTL (only the usual first-probe `member status: Removed` during startup) |
|
|
| New CLI | `--version` = `2b9a7c3c`; `instance list` → 4 rows; groups vs old: `+shared-schema +flow-debug` |
|
|
|
|
### Observed, not a regression
|
|
|
|
- Boot `[WRN] Startup configuration warning: Inbound API keys use the per-node SQLite store on a
|
|
2-node central cluster` — **spurious on this box.** `StartupValidator.CollectWarnings` counts
|
|
`Cluster:SeedNodes`, and vd03 carries the phantom second seed (8091/8092) that the same validator
|
|
requires (`>= 2` seed nodes, unconditionally). The node is single-member; nothing to act on until
|
|
the SqlServer key-store cutover.
|
|
- The CLI refuses to run without `--username/--password` (or `SCADABRIDGE_USERNAME/_PASSWORD`) even
|
|
though the server ignores them under DisableLogin — pass any value.
|
|
- The site's known `SiteLocalDbHealthCheck replicationConnected:false` (unreplicated single node) is
|
|
unchanged and cosmetic.
|
|
|
|
## Rollback
|
|
|
|
Stop Site, stop Central. `RESTORE DATABASE [ScadaBridge] FROM DISK =
|
|
'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-2b9a7c3c_20260910T104349.bak' WITH REPLACE`
|
|
(**required** — the protected protocol row is unreadable by `afb27e77` and the migration `Down`
|
|
throws over it). Rename `app` → `app.failed`, `app.bak.20260910T104529` → `app`, same for `cli`.
|
|
Start Central, then Site. Service env untouched; `.reg` exports are there if a service is ever
|
|
re-created.
|
|
|
|
Staging left on the box under `E:\psrem-stage\2b9a7c3c\` (zip, unpacked tree, migration SQL, swap
|
|
script, before/after snapshot JSON). The `_appsettings-keep-20260910T104529` copy and the
|
|
`*.bak.20260910T104529` directories can go once the build has soaked.
|