diff --git a/docs/deployment/deploy-prep/2026-11-15-143/README.md b/docs/deployment/deploy-prep/2026-11-15-143/README.md new file mode 100644 index 00000000..9ff91117 --- /dev/null +++ b/docs/deployment/deploy-prep/2026-11-15-143/README.md @@ -0,0 +1,76 @@ +# Deploy pack: ScadaBridge#143, hash-chain removal (`8e32f353` → `061432c8`), DEV vd03 then QA vq03/vq04 + +**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". Deploy current `main` to DEV, then QA. The #116 QA site-pair drill follows QA **only if** every QA CvdReactor instance is idle. + +## Build +`main` @ `061432c8`. Host and CLI are published with `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false` (the boxes have no .NET 10). The app has 1056 files and the cli 224. The pins are `ZB.MOM.WW.LocalDb 0.3.8` and `ZB.MOM.WW.Theme 1.0.1`. + +## Delta `8e32f353..061432c8` (25 commits) +- **#143 / #107:** the AuditLog hash chain, head export and two-login maintenance hardening are removed (`ed2ed0a3`, `b418408c`, merge `3dea146c`). The archive is kept at format v2. A retired `AuditLog:HashChain*` / `MaintenanceConnectionString` key is ignored and logs one warning. +- **#141:** the DCL fails over within seconds when the connected OPC UA server is going away (`c4ec51e8`). +- **#105:** the legacy site-psk-invalidation topic is removed (`89d2becb`). +- **UI consistency:** Theme 1.0.0 → 1.0.1 (dark palette, reflow), dialog naming and focus containment, controls and editor errors, semantic tables, the audit-window validation and the dashboard count reads (`cdd3adf6`…`6489c9fc`). +- **Docs and deploy records only:** #52/#142, #116 drills, #107/#138 records. +- **No appsettings change.** The only option change is a removal (`AuditLog:HashChain`). + +## Migration 64 → 65 +- `migrate-64-to-65.idempotent.sql` was generated at `061432c8` with `dotnet ef migrations script 20261110120000_AuditMaintenanceGrants --idempotent`. Its SHA-256 is `a16a150de1e1f6c4cb9153e1c53bed95a029d656f18c3dc3b61ec6575198f7a7`, and with comments stripped it is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql`. +- It applies one migration, `20261115221558_RemoveAuditLogHashChain`, in one transaction, and every statement is existence-guarded. +- It is run with `SET XACT_ABORT ON` and `sqlcmd -I -b`. On DEV it runs on vd03; on QA it runs on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`). The login is the Central service's own connection string, with the password only in `SQLCMDPASSWORD`. +- **Gate:** `8e32f353` maps `ChainSeq` / `RowHash`, so it cannot run on schema 65. The migration therefore runs with every ScadaBridge service on the tier stopped, and only the new binaries start afterwards. + +## Env change (Central service only) +Remove `AuditLog__HashChain__Enabled` and any other `AuditLog__HashChain__*` / `AuditLog__MaintenanceConnectionString` entries. Every other entry must stay byte-identical, and this is proven by a digest (name + SHA-256 prefix, values never printed). On QA the per-node `JwtExpiryMinutes` / `IdleTimeoutMinutes` drift sits in appsettings and is kept. + +## Order per tier +1. **Preflight**, read-only: + - live Host is `8e32f353`; + - 64 migrations; + - the chain objects are present: 2 columns, 2 tables, 2 indexes, the `_chainer` role; + - the env holds `AuditLog__HashChain__Enabled=true`; + - health and the active node. +2. **Backup:** `COPY_ONLY, CHECKSUM, COMPRESSION` + `RESTORE VERIFYONLY`; `reg export` of the service keys; appsettings kept with hashes. +3. **Stop:** DEV stops Site, then Central. QA stops both Sites, then both Centrals, so the pair goes down together. +4. **Migrate** 64 → 65. +5. **Swap:** `app` / `cli` → `*.bak.`, copy in the staged build, restore the kept appsettings (hash match). Re-assert `failureflag 1`. +6. **Env:** remove the retired keys from the Central env. +7. **Start:** Central(s) first, then Site(s). On QA vq03 goes first, then vq04. + +## Gates +- `/health/ready` 200 on every service, with one active node per pair. +- ProductVersion `+061432c8` on Host and CLI. +- 65 migrations. +- `ChainSeq` / `RowHash`, `AuditLogChainHead` / `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq` / `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` are **absent**; `scadabridge_audit_purger` and `_writer` are present. +- New AuditLog rows are written after the start. +- No `no longer supported` (retired-key) warning in the Central log. +- A bogus inbound key gets 401. +- Data connections are at baseline. On QA: LocalDb sync connected, Z28062–Z28069 at 244 Good, WarmCanary connected. +- OTLP `service.version` is `+061432c8`. +- Overview is Up. +- The CLI `audit` group has no `verify-chain` / `verify-heads`. + +## Rollback +Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore: +1. Stop all services. +2. `RESTORE … WITH REPLACE, CHECKSUM` from the pre-window backup. +3. `app` / `cli` ← `*.bak.`. +4. `reg import` the pre-window Central key, which puts `AuditLog__HashChain__Enabled=true` back. +5. Start the services: on QA the pair starts together. + +Never roll back past `0e540291`. The swap scripts auto-roll-back if the start fails. Once the migration has run, that auto-rollback restores the DB. + +## #116 site-pair drill (QA, conditional) +Two snapshots about 60 s apart are taken on every QA CvdReactor instance. If any instance has a cycle running, a cycle ID, FurnaceTemp ≥ 60 or a `RecipeDownloadFlag` True, the drill is **skipped**. + +If every instance is idle: +1. A kill drill of the active `ScadaBridge-Site` process, picked by service PID. +2. After about 2 min stable, a stop-and-hold drill: disable the SCM restart, kill, hold ≥ 60 s, capture `[leader-auto-down] DOWNING`, restore the recovery config byte-identical, then start. + +The gates afterwards are: instances re-hosted at 244 Good, one active site, LocalDb sync connected, and auto-restart restored. + +## Scripts (as run) +- `dev-deploy.ps1`: DEV, in one call. +- `qa.tmpl.ps1`: QA, rendered per step with `__ACTION__` / `__TS__`. The actions are `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify` and `rollback-swap`. +- `drill.tmpl.ps1`: the #116 site drill helper, with actions `status`, `kill`, `disable`, `kill-held`, `restore` and `start`. + +Results are in `deployment-records/2026-11-15-wonder-app-vd03-061432c8.md` and `…-vq03-vq04-061432c8.md`. diff --git a/docs/deployment/deploy-prep/2026-11-15-143/dev-deploy.ps1 b/docs/deployment/deploy-prep/2026-11-15-143/dev-deploy.ps1 new file mode 100644 index 00000000..89fa7a96 --- /dev/null +++ b/docs/deployment/deploy-prep/2026-11-15-143/dev-deploy.ps1 @@ -0,0 +1,182 @@ +# ScadaBridge DEV (wonder-app-vd03) 8e32f353 -> 061432c8 : ScadaBridge#143 (hash-chain removal), migration 64 -> 65 +# winrm --timeout 1200 ps vd03 -f dev-deploy.ps1 +# Single-node box: ONE central + ONE site; DB on this host. +# Order: preflight gates -> reg export -> DB backup (COPY_ONLY,CHECKSUM,COMPRESSION + VERIFYONLY) -> keep appsettings +# -> stop Site, Central -> migrate 64->65 (XACT_ABORT, sqlcmd -I -b) -> failureflag -> swap app/cli -> restore appsettings +# -> remove retired AuditLog env keys from Central -> start Central, Site -> post checks. +# On failure after the stop: restore DB (if migrated) + old binaries + Central env (.reg), restart. Values are never printed. +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' + +$sha = '061432c8' +$fromSha = '8e32f353' +$zipSha256 = 'B6E85ECC24186AF8B658FA6F1E38F080588EA5F78DD856A4356E696897FF4095' +$sqlSha256 = 'A16A150DE1E1F6C4CB9153E1C53BED95A029D656F18C3DC3B61EC6575198F7A7' +$stageDir = "E:\ApiInstall\_stage-sb-$sha" +$stage = "$stageDir\unpacked" +$root = 'E:\ApiInstall\ScadaBridge' +$bakDir = 'E:\ApiInstall\ScadaBridge-backups' +$ts = Get-Date -Format 'yyyyMMddTHHmmss' +$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt" +$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json' +$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central' +function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1}" -f (Get-Date), $m; $line; Add-Content -Path $log -Value $line } +function Is-Retired($e) { $e -like 'AuditLog__HashChain__*' -or $e -like 'AuditLog__MaintenanceConnectionString=*' } +function Digest($entries) { + $h = [System.Security.Cryptography.SHA256]::Create(); $o = @() + foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += ''; continue } + $v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') }) + $o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) } + $h.Dispose(); $o | Sort-Object +} +function Wait-Ready($url, $seconds) { + $deadline = (Get-Date).AddSeconds($seconds) + while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 } + return $false +} +function Stop-Both { + foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') { + if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') } + Say "$s stopped" + } + $deadline = (Get-Date).AddSeconds(30) + while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 } + if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running after stop" } + Say "no Host.exe process remains" +} +function Start-Both { + Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central' + $ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok" + if (-not $ok) { throw "central did not become ready" } + Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site' + $ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok" + if (-not $ok) { throw "site did not become ready" } +} + +Say "ScadaBridge#143 deploy $fromSha -> $sha on $env:COMPUTERNAME" + +# ---- preflight gates (nothing changed yet) ---------------------------------------------------- +if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" } +if ((Get-FileHash "$stageDir\migrate-64-to-65.idempotent.sql" -Algorithm SHA256).Hash -ne $sqlSha256) { throw "sql hash mismatch" } +if (Test-Path $stage) { Remove-Item $stage -Recurse -Force } +Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage +$stagedVer = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion +$stagedCli = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion +if ($stagedVer -notlike "*+$sha*" -or $stagedCli -notlike "*+$sha*") { throw "staged exe is $stagedVer / $stagedCli" } +$liveVer = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion +if ($liveVer -notlike "*+$fromSha*") { throw "live exe is $liveVer, not $fromSha" } +Say "staged $stagedVer; live $liveVer" +Say ("staged app files {0}, cli files {1}" -f (Get-ChildItem "$stage\app" -Recurse -File).Count, (Get-ChildItem "$stage\cli" -Recurse -File).Count) + +$envBefore = @((Get-ItemProperty $centralKey).Environment) +$retired = @($envBefore | Where-Object { Is-Retired $_ }) +Say ("central env entries {0}; retired keys present: {1}" -f $envBefore.Count, (($retired | % { $_.Substring(0,$_.IndexOf('=')) }) -join ',')) +if (-not ($envBefore -contains 'AuditLog__HashChain__Enabled=true')) { throw "AuditLog__HashChain__Enabled=true not found in the Central env" } + +$cs = ($envBefore | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=','' +$sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs +$db = $sb['Database']; if (-not $db) { $db = $sb.InitialCatalog } +if ($db -ne 'ScadaBridge') { throw "unexpected catalog" } +$env:SQLCMDPASSWORD = $sb.Password +function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed: $q" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() } +$chainQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',COL_LENGTH('dbo.AuditLog','ChainSeq'),'|',COL_LENGTH('dbo.AuditLog','RowHash'),'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainHead','U') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainTombstone','U') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID('dbo.AuditLog') AND name IN ('IX_AuditLog_ChainSeq','IX_AuditLog_Unchained')),'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog))" +$pre = Sql-Scalar $chainQ +Say "schema before (migrations|ChainSeq|RowHash|Head|Tomb|chainIdx|chainer|purger|writer|rows): $pre" +if (-not $pre.StartsWith('64|')) { throw "expected 64 migrations: $pre" } + +# ---- backups (services still running) --------------------------------------------------------- +foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { + & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null + if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" } + Say "reg export $s -> $stageDir\_envbak-$s-$ts.reg" +} +New-Item -ItemType Directory -Force -Path $bakDir | Out-Null +$bak = "$bakDir\ScadaBridge_pre-$($sha)_$ts.bak" +& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null +if ($LASTEXITCODE -ne 0) { throw "backup failed" } +& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null +if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" } +Say "db backup $bak ($((Get-Item $bak).Length) bytes, VERIFYONLY ok)" +$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts" +foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) } + +# ---- outage window ------------------------------------------------------------------------- +$swapped = $false; $migrated = $false; $envChanged = $false +try { + Stop-Both + + $migSql = "$stageDir\_migrate-xact-$ts.sql" + # one UTF-8 file WITHOUT any BOM (a BOM after a prepended header broke sqlcmd in #126) + $body = [IO.File]::ReadAllText("$stageDir\migrate-64-to-65.idempotent.sql", [Text.Encoding]::UTF8).TrimStart([char]0xFEFF) + [IO.File]::WriteAllText($migSql, "SET XACT_ABORT ON;`r`nGO`r`n" + $body, (New-Object System.Text.UTF8Encoding $false)) + Say "migrating 64 -> 65" + $migrated = $true + $out = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -I -b -i $migSql 2>&1 + $out | ForEach-Object { Say " sql: $_" } + if ($LASTEXITCODE -ne 0) { throw "migration failed (exit $LASTEXITCODE)" } + $post = Sql-Scalar $chainQ + Say "schema after: $post" + if (-not $post.StartsWith('65|')) { throw "expected 65 migrations after migrate: $post" } + + foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { + & sc.exe failureflag $s 1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failureflag $s failed" } + } + Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"; $swapped = $true + Say "renamed app/cli -> *.bak.$ts" + Copy-Item "$stage\app" "$root\app" -Recurse + Copy-Item "$stage\cli" "$root\cli" -Recurse + foreach ($f in $settings) { + Copy-Item "$keep\$f" "$root\app\$f" -Force + if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings restore mismatch: $f" } + Say "restored $f (sha256 match)" + } + Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion) + + $keepEnv = @($envBefore | Where-Object { -not (Is-Retired $_) }) + Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString; $envChanged = $true + $envAfter = @((Get-ItemProperty $centralKey).Environment) + foreach ($e in $keepEnv) { if ($envAfter -notcontains $e) { throw "an existing env entry changed" } } + if (@($envAfter | Where-Object { Is-Retired $_ }).Count -ne 0) { throw "retired env key still present" } + Say ("central env entries {0} -> {1}" -f $envBefore.Count, $envAfter.Count) + Digest $envAfter | ForEach-Object { Say " env $_" } + + Start-Both +} +catch { + Say "ERROR: $($_.Exception.Message)" + Say "AUTO-ROLLBACK (migrated=$migrated swapped=$swapped envChanged=$envChanged)" + try { + Stop-Both + if ($migrated) { + & sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "ALTER DATABASE [$db] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [$db] FROM DISK = N'$bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [$db] SET MULTI_USER;" | ForEach-Object { Say " restore: $_" } + if ($LASTEXITCODE -ne 0) { throw "DB restore failed" } + Say "DB restored from $bak" + } + if ($swapped) { + if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" } + if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" } + Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli' + Say "restored app/cli from *.bak.$ts" + } + if ($envChanged) { Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$envBefore) -Type MultiString; Say "central env restored" } + Start-Both + Say ("rolled back; live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion) + } catch { Say "AUTO-ROLLBACK FAILED: $($_.Exception.Message) - manual action needed" } + Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue + exit 1 +} + +# ---- post checks (verify of this deploy) ------------------------------------------------------ +Start-Sleep 20 +$q2 = Sql-Scalar "SELECT CONCAT(COUNT_BIG(*),'|',CONVERT(varchar(33),MAX(OccurredAtUtc),126)) FROM dbo.AuditLog" +Say "auditlog rows|newest: $q2" +$logDir = "$root\central\logs" +if (Test-Path $logDir) { + $f = Get-ChildItem $logDir -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1 + $hits = @(Select-String -Path $f.FullName -Pattern 'no longer supported' -SimpleMatch) + Say ("central log {0}: retired-key warnings since start = {1}" -f $f.Name, $hits.Count) +} +foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { Say ("$s " + (Get-Service $s).Status + " / " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim()) } +Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue +Say "DEPLOY DONE ts=$ts backup=$bak" +exit 0 diff --git a/docs/deployment/deploy-prep/2026-11-15-143/drill.tmpl.ps1 b/docs/deployment/deploy-prep/2026-11-15-143/drill.tmpl.ps1 new file mode 100644 index 00000000..ec14b271 --- /dev/null +++ b/docs/deployment/deploy-prep/2026-11-15-143/drill.tmpl.ps1 @@ -0,0 +1,27 @@ +# #116 QA site-pair drill helper. Rendered per call: ACTION. Service ScadaBridge-Site only. +$ErrorActionPreference = 'Stop' +$Action = '__ACTION__' +$svc = 'ScadaBridge-Site' +function Say($m) { "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m } +function Rec { Say ("qfailure: " + ((& sc.exe qfailure $svc | Out-String) -replace '\s+',' ').Trim()); Say ("qfailureflag: " + ((& sc.exe qfailureflag $svc | Out-String) -replace '\s+',' ').Trim()) } +$pidNow = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId +switch ($Action) { + 'status' { Say "$svc state $((Get-Service $svc).Status) pid $pidNow"; Rec } + 'kill' { + if (-not $pidNow -or $pidNow -eq 0) { throw "no running pid for $svc" } + $p = Get-Process -Id $pidNow; if ($p.ProcessName -ne 'ZB.MOM.WW.ScadaBridge.Host') { throw "pid $pidNow is $($p.ProcessName)" } + Say "KILL $svc pid $pidNow"; Stop-Process -Id $pidNow -Force; Say "killed" } + 'disable' { + & sc.exe failure $svc reset= 86400 actions= none/0/none/0/none/0 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failure disable" } + $q = (& sc.exe qfailure $svc | Out-String); if ($q -match 'RESTART') { throw "restart action still present" }; Rec } + 'kill-held' { + $q = (& sc.exe qfailure $svc | Out-String); if ($q -match 'RESTART') { throw "refusing: restart action present" } + $p = Get-Process -Id $pidNow; if ($p.ProcessName -ne 'ZB.MOM.WW.ScadaBridge.Host') { throw "pid $pidNow is $($p.ProcessName)" } + Say "KILL-HELD $svc pid $pidNow"; Stop-Process -Id $pidNow -Force; Say "killed (no auto-restart)" } + 'restore' { + & sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failure restore" } + & sc.exe failureflag $svc 1 | Out-Null; Rec } + 'start' { Say "starting $svc"; Start-Service $svc; (Get-Service $svc).WaitForStatus('Running','00:01:00'); Say "running pid $((Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId)" } + default { throw "bad action" } +} +exit 0 diff --git a/docs/deployment/deploy-prep/2026-11-15-143/migrate-64-to-65.idempotent.sql b/docs/deployment/deploy-prep/2026-11-15-143/migrate-64-to-65.idempotent.sql new file mode 100644 index 00000000..da3a8d34 --- /dev/null +++ b/docs/deployment/deploy-prep/2026-11-15-143/migrate-64-to-65.idempotent.sql @@ -0,0 +1,86 @@ +BEGIN TRANSACTION; +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + + IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_ChainSeq' AND object_id = OBJECT_ID('dbo.AuditLog')) + EXEC(N'DROP INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog;'); + IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_Unchained' AND object_id = OBJECT_ID('dbo.AuditLog')) + EXEC(N'DROP INDEX IX_AuditLog_Unchained ON dbo.AuditLog;'); +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + + IF DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NOT NULL + BEGIN + IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL + EXEC(N'REVOKE UPDATE (ChainSeq, RowHash) ON dbo.AuditLog FROM scadabridge_audit_chainer;'); + EXEC(N'REVOKE SELECT ON dbo.AuditLog FROM scadabridge_audit_chainer;'); + + DECLARE @dropMembers nvarchar(max) = N''; + SELECT @dropMembers = @dropMembers + N'ALTER ROLE scadabridge_audit_chainer DROP MEMBER ' + QUOTENAME(m.name) + N';' + CHAR(10) + FROM sys.database_role_members rm + JOIN sys.database_principals r ON r.principal_id = rm.role_principal_id + JOIN sys.database_principals m ON m.principal_id = rm.member_principal_id + WHERE r.name = N'scadabridge_audit_chainer'; + EXEC sp_executesql @dropMembers; + + EXEC(N'DROP ROLE scadabridge_audit_chainer;'); + END +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + + IF OBJECT_ID('dbo.AuditLogChainHead', 'U') IS NOT NULL + EXEC(N'DROP TABLE dbo.AuditLogChainHead;'); + IF OBJECT_ID('dbo.AuditLogChainTombstone', 'U') IS NOT NULL + EXEC(N'DROP TABLE dbo.AuditLogChainTombstone;'); +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + + IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL + EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN ChainSeq;'); + IF COL_LENGTH('dbo.AuditLog', 'RowHash') IS NOT NULL + EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN RowHash;'); +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + + IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NOT NULL + BEGIN + EXEC(N'REVOKE ALTER ANY DATASPACE FROM scadabridge_audit_purger;'); + EXEC(N'REVOKE UPDATE (SourceNode) ON dbo.AuditLog FROM scadabridge_audit_purger;'); + END +END; + +IF NOT EXISTS ( + SELECT * FROM [__EFMigrationsHistory] + WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain' +) +BEGIN + INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) + VALUES (N'20261115221558_RemoveAuditLogHashChain', N'10.0.13'); +END; + +COMMIT; +GO + diff --git a/docs/deployment/deploy-prep/2026-11-15-143/qa.tmpl.ps1 b/docs/deployment/deploy-prep/2026-11-15-143/qa.tmpl.ps1 new file mode 100644 index 00000000..1161e961 --- /dev/null +++ b/docs/deployment/deploy-prep/2026-11-15-143/qa.tmpl.ps1 @@ -0,0 +1,155 @@ +# ScadaBridge QA (vq03/vq04) 8e32f353 -> 061432c8 : ScadaBridge#143. Rendered per call: ACTION / TS. +# prep | backup (vq03) | stop-site | stop-central | migrate (vq03) | swap | start-central | start-site | verify | rollback-swap +$ErrorActionPreference = 'Stop' +$ProgressPreference = 'SilentlyContinue' +$Action = '__ACTION__' +$ts = '__TS__' +$sha = '061432c8' +$fromSha = '8e32f353' +$zipSha256 = 'B6E85ECC24186AF8B658FA6F1E38F080588EA5F78DD856A4356E696897FF4095' +$sqlSha256 = 'A16A150DE1E1F6C4CB9153E1C53BED95A029D656F18C3DC3B61EC6575198F7A7' +$stageDir = "E:\psrem-stage\sb-$sha" +$stage = "$stageDir\unpacked" +$root = 'E:\ApiInstall\ScadaBridge' +$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt" +$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json' +$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central' +$bak = "F:\MSSQL\BACKUP\ScadaBridge_pre-$($sha)_$ts.bak" +function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m; $line; Add-Content -Path $log -Value $line } +function Is-Retired($e) { $e -like 'AuditLog__HashChain__*' -or $e -like 'AuditLog__MaintenanceConnectionString=*' } +function Digest($entries) { + $h = [System.Security.Cryptography.SHA256]::Create(); $o = @() + foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += ''; continue } + $v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') }) + $o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) } + $h.Dispose(); $o | Sort-Object +} +function Wait-Ready($url, $seconds) { + $deadline = (Get-Date).AddSeconds($seconds) + while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 } + return $false +} +function Stop-Svc($s) { + if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') } + Say "$s stopped" +} +function Sql-Init { + $envC = @((Get-ItemProperty $centralKey).Environment) + $cs = ($envC | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=','' + $script:sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs + $script:db = $sb['Database']; if (-not $db) { $script:db = $sb.InitialCatalog } + if ($db -ne 'ScadaBridge') { throw "unexpected catalog" } + $env:SQLCMDPASSWORD = $sb.Password +} +function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() } +$chainQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',COL_LENGTH('dbo.AuditLog','ChainSeq'),'|',COL_LENGTH('dbo.AuditLog','RowHash'),'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainHead','U') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainTombstone','U') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID('dbo.AuditLog') AND name IN ('IX_AuditLog_ChainSeq','IX_AuditLog_Unchained')),'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog),'|',CONVERT(varchar(33),(SELECT MAX(OccurredAtUtc) FROM dbo.AuditLog),126))" + +switch ($Action) { +'prep' { + Say "PREP $fromSha -> $sha" + if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" } + if (Test-Path $stage) { Remove-Item $stage -Recurse -Force } + Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage + $sv = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + $sc = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion + if ($sv -notlike "*+$sha*" -or $sc -notlike "*+$sha*") { throw "staged exe $sv / $sc" } + $lv = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + if ($lv -notlike "*+$fromSha*") { throw "live exe $lv" } + Say "staged $sv; live $lv; app files $((Get-ChildItem "$stage\app" -Recurse -File).Count), cli $((Get-ChildItem "$stage\cli" -Recurse -File).Count)" + foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { + & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null + if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" } + Say "reg export $s" + $e = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment) + Say ("$s env entries {0}" -f $e.Count); Digest $e | % { Say " env $s $_" } + Say ("recovery ${s}: " + ((& sc.exe qfailure $s | Out-String) -replace '\s+',' ').Trim()) + Say ("recovery ${s}: " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim()) + } + $envC = @((Get-ItemProperty $centralKey).Environment) + if (-not ($envC -contains 'AuditLog__HashChain__Enabled=true')) { throw "AuditLog__HashChain__Enabled=true not in Central env" } + Say ("retired keys: " + ((@($envC | ? { Is-Retired $_ }) | % { $_.Substring(0,$_.IndexOf('=')) }) -join ',')) + $keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts" + foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) } +} +'backup' { + Sql-Init + if ((Get-FileHash "$stageDir\migrate-64-to-65.idempotent.sql" -Algorithm SHA256).Hash -ne $sqlSha256) { throw "sql hash mismatch" } + $pre = Sql-Scalar $chainQ; Say "schema before: $pre" + if (-not $pre.StartsWith('64|')) { throw "expected 64 migrations" } + & sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null + if ($LASTEXITCODE -ne 0) { throw "backup failed" } + & sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null + if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" } + Say "backup $bak VERIFYONLY ok" +} +'stop-site' { Stop-Svc 'ScadaBridge-Site' } +'stop-central' { + Stop-Svc 'ScadaBridge-Central' + $deadline = (Get-Date).AddSeconds(30) + while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 } + if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running" } + Say "no Host.exe remains" +} +'migrate' { + Sql-Init + foreach ($h in 'wonder-app-vq03','wonder-app-vq04') { foreach ($p in 8085,8084) { try { Invoke-WebRequest -UseBasicParsing "http://$($h):$p/health/ready" -TimeoutSec 3 | Out-Null; throw "$h`:$p still answering - not all services stopped" } catch [System.Net.WebException] {} } } + $migSql = "$stageDir\_migrate-xact-$ts.sql" + $body = [IO.File]::ReadAllText("$stageDir\migrate-64-to-65.idempotent.sql", [Text.Encoding]::UTF8).TrimStart([char]0xFEFF) + [IO.File]::WriteAllText($migSql, "SET XACT_ABORT ON;`r`nGO`r`n" + $body, (New-Object System.Text.UTF8Encoding $false)) + Say "migrating 64 -> 65" + $out = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -I -b -i $migSql 2>&1 + $rc = $LASTEXITCODE + $out | % { Say " sql: $_" } + if ($rc -ne 0) { throw "migration failed (exit $rc)" } + $post = Sql-Scalar $chainQ; Say "schema after: $post" + if (-not $post.StartsWith('65|')) { throw "expected 65 migrations" } +} +'swap' { + if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe running" } + foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { & sc.exe failureflag $s 1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "failureflag $s" } } + Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts" + Say "renamed app/cli -> *.bak.$ts" + Copy-Item "$stage\app" "$root\app" -Recurse + Copy-Item "$stage\cli" "$root\cli" -Recurse + $keep = "$root\_appsettings-keep-$ts" + foreach ($f in $settings) { + Copy-Item "$keep\$f" "$root\app\$f" -Force + if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings mismatch $f" } + Say "restored $f (sha256 match)" + } + Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion) + $before = @((Get-ItemProperty $centralKey).Environment) + $keepEnv = @($before | Where-Object { -not (Is-Retired $_) }) + Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString + $after = @((Get-ItemProperty $centralKey).Environment) + foreach ($e in $keepEnv) { if ($after -notcontains $e) { throw "env entry changed" } } + if (@($after | ? { Is-Retired $_ }).Count -ne 0) { throw "retired key still present" } + Say ("central env entries {0} -> {1}" -f $before.Count, $after.Count); Digest $after | % { Say " env ScadaBridge-Central $_" } +} +'start-central' { Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'; $ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"; if (-not $ok) { throw "central not ready" } } +'start-site' { Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'; $ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"; if (-not $ok) { throw "site not ready" } } +'verify' { + foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { + Say ("$s " + (Get-Service $s).Status) + Say ("recovery ${s}: " + ((& sc.exe qfailure $s | Out-String) -replace '\s+',' ').Trim()) + Say ("recovery ${s}: " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim()) + Digest @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment) | % { Say " env $s $_" } + } + $logDir = "$root\central\logs" + $f = Get-ChildItem $logDir -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1 + Say ("central log {0}: retired-key warnings = {1}" -f $f.Name, @(Select-String -Path $f.FullName -Pattern 'no longer supported' -SimpleMatch).Count) + if ($env:COMPUTERNAME -eq 'WONDER-APP-VQ03') { Sql-Init; Say ("schema now: " + (Sql-Scalar $chainQ)) } +} +'rollback-swap' { + foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') { Stop-Svc $s } + if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" } + if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" } + Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli' + & reg.exe import "$stageDir\_envbak-ScadaBridge-Central-$ts.reg" | Out-Null + Say "rolled back binaries + Central env (services left STOPPED; restore DB next)" +} +default { throw "bad action $Action" } +} +Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue +Say "$Action OK" +exit 0 diff --git a/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md b/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md new file mode 100644 index 00000000..f8f3ee96 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md @@ -0,0 +1,68 @@ +# 2026-11-15: DEV (wonder-app-vd03) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143) + +**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". IT security asked for the AuditLog hash chain to be removed. The pack is `docs/deployment/deploy-prep/2026-11-15-143/`. + +**Build:** `main` @ `061432c8`, ProductVersion `1.0.0+061432c8c4299505a45d4cf396c3d62139865080`. +- Host and CLI were published self-contained win-x64 from a clean detached worktree: 1056 app files and 224 cli files. +- The zip is 130 414 127 B, SHA-256 `b6e85ecc24186af8b658fa6f1e38f080588ea5f78dd856a4356e696897ff4095`. +- Pins: LocalDb 0.3.8, Theme 1.0.1. + +**Delta:** 25 commits. See the pack: the hash-chain, head-export and two-login removal (`3dea146c`), #141, #105, and the UI-consistency / Theme 1.0.x work. There is one migration, `20261115221558_RemoveAuditLogHashChain`. The script `migrate-64-to-65.idempotent.sql` has SHA-256 `a16a150d…98f7a7`, is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql` once comments are stripped, and was run with `SET XACT_ABORT ON` and `sqlcmd -I -b`. + +**Transport:** one deploy script over `winrm` (deploy scope, owner-authorized), `dev-deploy.ps1` in `E:\ApiInstall\_stage-sb-061432c8\`. Staging went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191258.txt`. + +**Not touched:** QA (see its own record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. Nothing was deployed to any instance. + +## Preflight (in the script, before anything changed) + +| Check | Result | +|---|---| +| Live build | `8e32f353` | +| Staged build | `061432c8` Host + CLI | +| Schema | 64 migrations; `ChainSeq` (8) / `RowHash` (32) present, both chain tables, 2 chain indexes, roles `_chainer`, `_purger`, `_writer`; AuditLog 2 853 rows | +| Central env | 13 entries; retired key present: `AuditLog__HashChain__Enabled` (= `true`) | +| appsettings | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` (unchanged since 11-13) | +| Health / baseline | central and site ready/active 200; Z28062 270 attrs, Good 144, 8 alarms | + +## Window + +| Step | Local (EST) | +|---|---| +| `reg export` of both service keys; backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak` (131 757 056 B, `COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); appsettings kept in `_appsettings-keep-20261115T191258` | 19:13:11 → 19:13:14 | +| Stop Site, then Central; no `Host.exe` left | 19:13:14 → 19:13:35 | +| Migrate 64 → 65 | 19:13:35 (< 1 s) | +| `failureflag 1` re-asserted; `app`/`cli` → `*.bak.20261115T191258`; new build in; appsettings restored (hash match) | 19:13:35 → 19:13:41 | +| Central env: `AuditLog__HashChain__Enabled` removed, 13 → 12 entries; the other 12 byte-identical (digest-checked) | 19:13:41 | +| Central ready, then Site ready | 19:13:53 / 19:14:02 | + +**Downtime:** the site data path was down for about 48 s (19:13:14 → 19:14:02) and the central API for about 29 s (19:13:25 → 19:13:54). + +## Gates (all pass) + +| Gate | Result | +|---|---| +| Build | Host + CLI `1.0.0+061432c8…` | +| Schema | **65** migrations; `ChainSeq`, `RowHash`, `AuditLogChainHead`, `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq`, `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` all **absent**; `scadabridge_audit_purger` + `_writer` present; 2 853 rows kept | +| Health | ready/active 200/200 (central `:8085`, site `:8084`). Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone). Site checks: `akka-cluster`, `localdb` Healthy | +| Retired-key warning | 0 in `scadabridge-central-20261115.log`, because the env is clean | +| AuditLog | new rows after the start: 2 `InboundAuthFailure` from `central-a` (the bogus-key probes) | +| Inbound API | bogus `X-API-Key` → **401** ×2 | +| MxGateway / instances | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / 270, 8 alarms (= baseline); 5 deployed instances | +| CLI | `audit` group: query, export, tree, backfill-source-node, archive; **no** `verify-chain` / `verify-heads` | +| OTLP | box collector `logs.jsonl` carries `service.version 1.0.0+061432c8…` (79 records at check time) | +| Overview `:5320` | 6/6 Up; ScadaBridge central Active/Up, leader chip, 4/4 checks | +| Recovery | both services `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` | +| Pending updates | 1 stale instance: Z28061, expected. Its machine moved to QA, and it was left on the old #52 body on purpose (#142 closed) | + +The env block still has `ScadaBridge__Database__MachineDataDb`, which has been unused since its removal on 2026-10-22. It is harmless and was not touched. + +## Rollback + +Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore: +1. Stop Site, then Central. +2. `RESTORE DATABASE [ScadaBridge] FROM DISK = N'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak' WITH REPLACE, CHECKSUM` (single-user first). +3. `app`/`cli` ← `*.bak.20261115T191258`. +4. `reg import E:\ApiInstall\_stage-sb-061432c8\_envbak-ScadaBridge-Central-20261115T191258.reg`, which puts `AuditLog__HashChain__Enabled=true` back. +5. Start Central, then Site. + +The restore loses every DB write since 19:13. The `*.bak.20261115T191258` and `_appsettings-keep-20261115T191258` folders can go once the build has soaked. diff --git a/docs/deployment/deployment-records/2026-11-15-wonder-app-vq03-vq04-061432c8.md b/docs/deployment/deployment-records/2026-11-15-wonder-app-vq03-vq04-061432c8.md new file mode 100644 index 00000000..3cb1c4f6 --- /dev/null +++ b/docs/deployment/deployment-records/2026-11-15-wonder-app-vq03-vq04-061432c8.md @@ -0,0 +1,116 @@ +# 2026-11-15: QA (wonder-app-vq03 + vq04) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143), #116 site-pair drills + +**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". The pack is `docs/deployment/deploy-prep/2026-11-15-143/`. DEV went first (`2026-11-15-wonder-app-vd03-061432c8.md`, all gates passed). + +**Build:** the same zip as DEV, `061432c8`, SHA-256 `b6e85ecc…ff4095`. It was pushed to `E:\psrem-stage\sb-061432c8\` on both boxes and its hash re-verified on each box: 1056 app files, 224 cli files. + +**Transport:** one rendered script per step over `winrm` (deploy scope, owner-authorized): `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify`. Staging and logs went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The backup and migration ran on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`), using the Central service's own connection string with the password in `SQLCMDPASSWORD` only. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191851.txt`. + +**Not touched:** DEV (separate record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector, `ScadaBridge-LDAP`. No instance was deployed to. + +## Preflight + +| Check | Result | +|---|---| +| Live / staged | `8e32f353` / `061432c8` on both boxes | +| Schema (wonder-sql-vq03) | 64 migrations; chain columns, tables, indexes and `_chainer` present; AuditLog 3 606 rows | +| Service env | Central 16 entries, Site 8 entries, **digest-identical on both boxes**; retired key `AuditLog__HashChain__Enabled` (`true`) on both Centrals | +| appsettings | `C0C653D6…` (both); Central vq03 `EF01B129…`, vq04 `F333D48F…`; Site vq03 `D0E202F9…`, vq04 `E9E6336F…`. These equal the 11-13 post-deploy hashes, and the per-node JWT/idle drift is kept | +| Recovery | all four: restart 5 s / 5 s / 10 s, reset 86 400, non-crash flag TRUE | +| Active | central vq03, site vq04 (since the #116 central drills earlier today) | +| Baseline | Z28061–Z28069 244/244 Good, 10 alarms each; WarmCanary 2/2 Good; MxGateway 2196/2196 good; LocalDb sync connected | + +## Window + +| Step | Local (EST) | +|---|---| +| Backup `F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak` (`COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); `reg export` + appsettings kept on both boxes in the prep step | 19:19:00 → 19:19:48 | +| Stop Site vq03 (standby) | 19:20:06 → 19:20:17 | +| Stop Site vq04 (active, **site data path down**) | 19:20:34 → 19:20:45 | +| Stop Central vq04 (standby) | 19:21:02 → 19:21:12 | +| Stop Central vq03 (active, **central API down**); no `Host.exe` on either box | 19:21:23 → 19:21:33 | +| Migrate 64 → 65 (it refuses to run while any ScadaBridge port still answers) | 19:21:54 (< 1 s) | +| Swap on both boxes in parallel: `failureflag 1`, `app`/`cli` → `*.bak.20261115T191851`, new build, appsettings restored (hash match), Central env 16 → 15 | 19:22:16 → 19:22:31 | +| Start Centrals (vq03, vq04 3 s later) → ready | 19:23:00 / 19:23:05 | +| Start Sites (vq03, vq04 3 s later) → ready | 19:23:25 / 19:23:30 | + +**Downtime:** the site data path was down for about 2 min 40 s (19:20:35 → 19:23:25) and the central API for about 1 min 37 s (19:21:23 → 19:23:00). Most of the gaps between steps are the ~15 s WinRM session set-up per call. + +## Gates (all pass) + +| Gate | Result | +|---|---| +| Build | Host + CLI `1.0.0+061432c8…` on both boxes | +| Schema | **65** migrations; `ChainSeq`, `RowHash`, both chain tables, both chain indexes and `scadabridge_audit_chainer` **absent**; `_purger` + `_writer` present; 3 606 rows kept | +| Health | ready 200 ×4; active: central vq03 200 / vq04 503, site vq03 200 / vq04 503. Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone) | +| Env | Central env = the pre-window entries minus `AuditLog__HashChain__Enabled`, byte-identical otherwise, identical on both boxes; Site env unchanged | +| Retired-key warning | 0 on both Centrals | +| Cluster / LocalDb | central Primary vq03 / Standby vq04, both online; site LocalDb `replicationConnected: true`, backlog 0 on both nodes | +| Inbound API | bogus key → **401** on vq03 and vq04 (through the standby relay); 2 new `InboundAuthFailure` rows from `central-a` | +| API keys | `MESTestKey`, `MES_Key`, `delmia-notifier-qa` | +| Data | MxGateway 2196/2196 good; WarmCanary connected, 2/2 good; Z28061–Z28069 244/244 Good, 10 alarms each (= baseline); 10 deployed instances | +| CLI | `audit` group has no `verify-chain` / `verify-heads` | +| OTLP | box collectors carry `service.version 1.0.0+061432c8…` (vq03 95, vq04 63 log records at check time) | +| Overview `:5320` | 12/12 Up on vq03 and vq04; ScadaBridge 4 instances / 2 clusters, central leader vq03 | +| Recovery | unchanged on all four services | +| Pending updates | 1 stale: **WarmCanary** (16). Its OPC UA connection config was edited after its 2026-11-13 deploy (#138 work). This deploy did not cause it, and it was left alone | + +**Purge/partition sanity:** skipped. There is no read-only way to exercise a purge or switch-out, and nothing is due: archive and retention are off. + +## #116 site-pair drills (owner condition: only if idle) + +**Idle check:** two snapshots of all nine CvdReactors (Z28061–Z28069), at 00:27:18Z and 00:28:38Z. +- Every side showed `CycleRunning` False (Good) and `MachineCycleID` 0. +- FurnaceTemp was 23–25 °C, and no `RecipeDownloadFlag` was True. +- Between the snapshots the only changes were ±1 argon purge flow (Z28061, Z28067) and ±1 °C (Z28065). + +So the drills ran. ZIPSEN is disabled and hosts no attributes. + +**Kill drill.** The active Site, vq03 (PID 8896, picked through the service), was killed at 00:29:06.0Z: + +| Event | UTC | After kill | +|---|---|---| +| vq03 Site `/health` stops answering | 00:29:06.4 | +0.4 s | +| vq04 `New incarnation of existing member … is trying to join`; `[leader-auto-down] … already "Down"; not scheduling a Down`; vq04 gains leadership | 00:29:13.05 | +7.0 s | +| vq04 Site active 200 | 00:29:13.7 | **+7.7 s** | +| vq03 back as standby (ready 200 / active 503) | 00:29:24.3 | +18.3 s | + +As in the central kill drill, the SCM restarts the killed process after 5 s, before the 15 s downing timer runs out, so Akka downs the old incarnation itself. After the drill all 10 instances were 2198/2198 Good on vq04, and LocalDb was connected. + +**Stop-and-hold drill** on the new active Site, vq04 (PID 18956), after about 1 min 20 s stable: +1. Recovery config recorded: RESTART 5000/5000/10000, reset 86400, non-crash flag TRUE. +2. Set to no actions. The kill script refuses to run while any RESTART action remains. +3. Killed at 00:30:32.7Z. +4. Held down for 113 s. +5. Recovery restored **byte-identical** to the recorded config (diff-checked), then started. + +| Event | UTC | After kill | +|---|---|---| +| Kill vq04 Site | 00:30:32.72 | 0 | +| vq03 `[leader-auto-down] … 10.220.158.3:8082 is unreachable; it will be downed if still unreachable after "00:00:15"` | 00:30:44.86 | +12.1 s | +| vq03 `[WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8082: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8082) is leader` | 00:30:59.87 | **+27.2 s** | +| vq03 Site active 200 | 00:31:00.17 | **+27.4 s** | +| Recovery restored, `Start-Service` | 00:31:56 / 00:32:25 | — | +| vq04 `Welcome from …10.220.158.2:8082`; standby (ready 200 / active 503) | 00:32:27.8 / 00:32:28.2 | 2.7 s after the start | + +vq04 rejoined without the #132 shut-out path (no exit 75), because it started after it had been removed. + +**After the drills (all pass):** +- ready 200 ×4; +- one active per pair: central vq03 and site vq03; +- 10 instances at 2198/2198 Good; +- MxGateway 2196 good, WarmCanary 2 good; +- LocalDb connected, backlog 0, on both nodes; +- both Site services back on RESTART 5 s / 5 s / 10 s, reset 86 400, flag TRUE; +- Overview 12/12 Up. + +**What this proves on QA:** a killed site node fails over in about 8 s through the SCM-restart path. A site node that stays down is downed by `leader-auto-down` after 15 s unreachable, and the pair takes over in about 27 s. Both match the central drills (`2026-11-15-qa-116-central-drill.md`). + +## Rollback (deploy) + +Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore, with the pair stopped together: +1. On each box, run `qa.tmpl.ps1` (in the pack and in `E:\psrem-stage\sb-061432c8\`) rendered with `__ACTION__`=`rollback-swap` and `__TS__`=`20261115T191851`. It stops Site then Central, puts back `app`/`cli` ← `*.bak.20261115T191851` and re-imports the pre-window Central key (`_envbak-ScadaBridge-Central-20261115T191851.reg`). Services are left stopped. +2. With all four stopped, from vq03: `ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [ScadaBridge] SET MULTI_USER;`. This loses every DB write since 19:19. +3. Start both Centrals together, then both Sites. + +The `*.bak.20261115T191851` and `_appsettings-keep-20261115T191851` folders can go once the build has soaked.