deploy(#143): DEV vd03 + QA vq03/vq04 to 061432c8 — hash chain removed (migration 65), #116 QA site-pair drills
DEV: ~48 s site / ~29 s central outage; QA: pair stop/start, ~2m40s site / ~1m37s central. Chain schema + chainer role gone, env key removed, all gates pass on both tiers. #116 QA site drills (all reactors idle): kill → takeover +7.7 s; stop-and-hold → [leader-auto-down] DOWNING +27.2 s, takeover +27.4 s, recovery restored byte-identical. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# Deploy pack: ScadaBridge#143, hash-chain removal (`8e32f353` → `061432c8`), DEV vd03 then QA vq03/vq04
|
||||
|
||||
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". Deploy current `main` to DEV, then QA. The #116 QA site-pair drill follows QA **only if** every QA CvdReactor instance is idle.
|
||||
|
||||
## Build
|
||||
`main` @ `061432c8`. Host and CLI are published with `-c Release -r win-x64 --self-contained true -p:PublishSingleFile=false` (the boxes have no .NET 10). The app has 1056 files and the cli 224. The pins are `ZB.MOM.WW.LocalDb 0.3.8` and `ZB.MOM.WW.Theme 1.0.1`.
|
||||
|
||||
## Delta `8e32f353..061432c8` (25 commits)
|
||||
- **#143 / #107:** the AuditLog hash chain, head export and two-login maintenance hardening are removed (`ed2ed0a3`, `b418408c`, merge `3dea146c`). The archive is kept at format v2. A retired `AuditLog:HashChain*` / `MaintenanceConnectionString` key is ignored and logs one warning.
|
||||
- **#141:** the DCL fails over within seconds when the connected OPC UA server is going away (`c4ec51e8`).
|
||||
- **#105:** the legacy site-psk-invalidation topic is removed (`89d2becb`).
|
||||
- **UI consistency:** Theme 1.0.0 → 1.0.1 (dark palette, reflow), dialog naming and focus containment, controls and editor errors, semantic tables, the audit-window validation and the dashboard count reads (`cdd3adf6`…`6489c9fc`).
|
||||
- **Docs and deploy records only:** #52/#142, #116 drills, #107/#138 records.
|
||||
- **No appsettings change.** The only option change is a removal (`AuditLog:HashChain`).
|
||||
|
||||
## Migration 64 → 65
|
||||
- `migrate-64-to-65.idempotent.sql` was generated at `061432c8` with `dotnet ef migrations script 20261110120000_AuditMaintenanceGrants --idempotent`. Its SHA-256 is `a16a150de1e1f6c4cb9153e1c53bed95a029d656f18c3dc3b61ec6575198f7a7`, and with comments stripped it is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql`.
|
||||
- It applies one migration, `20261115221558_RemoveAuditLogHashChain`, in one transaction, and every statement is existence-guarded.
|
||||
- It is run with `SET XACT_ABORT ON` and `sqlcmd -I -b`. On DEV it runs on vd03; on QA it runs on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`). The login is the Central service's own connection string, with the password only in `SQLCMDPASSWORD`.
|
||||
- **Gate:** `8e32f353` maps `ChainSeq` / `RowHash`, so it cannot run on schema 65. The migration therefore runs with every ScadaBridge service on the tier stopped, and only the new binaries start afterwards.
|
||||
|
||||
## Env change (Central service only)
|
||||
Remove `AuditLog__HashChain__Enabled` and any other `AuditLog__HashChain__*` / `AuditLog__MaintenanceConnectionString` entries. Every other entry must stay byte-identical, and this is proven by a digest (name + SHA-256 prefix, values never printed). On QA the per-node `JwtExpiryMinutes` / `IdleTimeoutMinutes` drift sits in appsettings and is kept.
|
||||
|
||||
## Order per tier
|
||||
1. **Preflight**, read-only:
|
||||
- live Host is `8e32f353`;
|
||||
- 64 migrations;
|
||||
- the chain objects are present: 2 columns, 2 tables, 2 indexes, the `_chainer` role;
|
||||
- the env holds `AuditLog__HashChain__Enabled=true`;
|
||||
- health and the active node.
|
||||
2. **Backup:** `COPY_ONLY, CHECKSUM, COMPRESSION` + `RESTORE VERIFYONLY`; `reg export` of the service keys; appsettings kept with hashes.
|
||||
3. **Stop:** DEV stops Site, then Central. QA stops both Sites, then both Centrals, so the pair goes down together.
|
||||
4. **Migrate** 64 → 65.
|
||||
5. **Swap:** `app` / `cli` → `*.bak.<ts>`, copy in the staged build, restore the kept appsettings (hash match). Re-assert `failureflag 1`.
|
||||
6. **Env:** remove the retired keys from the Central env.
|
||||
7. **Start:** Central(s) first, then Site(s). On QA vq03 goes first, then vq04.
|
||||
|
||||
## Gates
|
||||
- `/health/ready` 200 on every service, with one active node per pair.
|
||||
- ProductVersion `+061432c8` on Host and CLI.
|
||||
- 65 migrations.
|
||||
- `ChainSeq` / `RowHash`, `AuditLogChainHead` / `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq` / `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` are **absent**; `scadabridge_audit_purger` and `_writer` are present.
|
||||
- New AuditLog rows are written after the start.
|
||||
- No `no longer supported` (retired-key) warning in the Central log.
|
||||
- A bogus inbound key gets 401.
|
||||
- Data connections are at baseline. On QA: LocalDb sync connected, Z28062–Z28069 at 244 Good, WarmCanary connected.
|
||||
- OTLP `service.version` is `+061432c8`.
|
||||
- Overview is Up.
|
||||
- The CLI `audit` group has no `verify-chain` / `verify-heads`.
|
||||
|
||||
## Rollback
|
||||
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore:
|
||||
1. Stop all services.
|
||||
2. `RESTORE … WITH REPLACE, CHECKSUM` from the pre-window backup.
|
||||
3. `app` / `cli` ← `*.bak.<ts>`.
|
||||
4. `reg import` the pre-window Central key, which puts `AuditLog__HashChain__Enabled=true` back.
|
||||
5. Start the services: on QA the pair starts together.
|
||||
|
||||
Never roll back past `0e540291`. The swap scripts auto-roll-back if the start fails. Once the migration has run, that auto-rollback restores the DB.
|
||||
|
||||
## #116 site-pair drill (QA, conditional)
|
||||
Two snapshots about 60 s apart are taken on every QA CvdReactor instance. If any instance has a cycle running, a cycle ID, FurnaceTemp ≥ 60 or a `RecipeDownloadFlag` True, the drill is **skipped**.
|
||||
|
||||
If every instance is idle:
|
||||
1. A kill drill of the active `ScadaBridge-Site` process, picked by service PID.
|
||||
2. After about 2 min stable, a stop-and-hold drill: disable the SCM restart, kill, hold ≥ 60 s, capture `[leader-auto-down] DOWNING`, restore the recovery config byte-identical, then start.
|
||||
|
||||
The gates afterwards are: instances re-hosted at 244 Good, one active site, LocalDb sync connected, and auto-restart restored.
|
||||
|
||||
## Scripts (as run)
|
||||
- `dev-deploy.ps1`: DEV, in one call.
|
||||
- `qa.tmpl.ps1`: QA, rendered per step with `__ACTION__` / `__TS__`. The actions are `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify` and `rollback-swap`.
|
||||
- `drill.tmpl.ps1`: the #116 site drill helper, with actions `status`, `kill`, `disable`, `kill-held`, `restore` and `start`.
|
||||
|
||||
Results are in `deployment-records/2026-11-15-wonder-app-vd03-061432c8.md` and `…-vq03-vq04-061432c8.md`.
|
||||
@@ -0,0 +1,182 @@
|
||||
# ScadaBridge DEV (wonder-app-vd03) 8e32f353 -> 061432c8 : ScadaBridge#143 (hash-chain removal), migration 64 -> 65
|
||||
# winrm --timeout 1200 ps vd03 -f dev-deploy.ps1
|
||||
# Single-node box: ONE central + ONE site; DB on this host.
|
||||
# Order: preflight gates -> reg export -> DB backup (COPY_ONLY,CHECKSUM,COMPRESSION + VERIFYONLY) -> keep appsettings
|
||||
# -> stop Site, Central -> migrate 64->65 (XACT_ABORT, sqlcmd -I -b) -> failureflag -> swap app/cli -> restore appsettings
|
||||
# -> remove retired AuditLog env keys from Central -> start Central, Site -> post checks.
|
||||
# On failure after the stop: restore DB (if migrated) + old binaries + Central env (.reg), restart. Values are never printed.
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
|
||||
$sha = '061432c8'
|
||||
$fromSha = '8e32f353'
|
||||
$zipSha256 = 'B6E85ECC24186AF8B658FA6F1E38F080588EA5F78DD856A4356E696897FF4095'
|
||||
$sqlSha256 = 'A16A150DE1E1F6C4CB9153E1C53BED95A029D656F18C3DC3B61EC6575198F7A7'
|
||||
$stageDir = "E:\ApiInstall\_stage-sb-$sha"
|
||||
$stage = "$stageDir\unpacked"
|
||||
$root = 'E:\ApiInstall\ScadaBridge'
|
||||
$bakDir = 'E:\ApiInstall\ScadaBridge-backups'
|
||||
$ts = Get-Date -Format 'yyyyMMddTHHmmss'
|
||||
$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt"
|
||||
$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json'
|
||||
$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central'
|
||||
function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1}" -f (Get-Date), $m; $line; Add-Content -Path $log -Value $line }
|
||||
function Is-Retired($e) { $e -like 'AuditLog__HashChain__*' -or $e -like 'AuditLog__MaintenanceConnectionString=*' }
|
||||
function Digest($entries) {
|
||||
$h = [System.Security.Cryptography.SHA256]::Create(); $o = @()
|
||||
foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += '<unparsable>'; continue }
|
||||
$v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') })
|
||||
$o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) }
|
||||
$h.Dispose(); $o | Sort-Object
|
||||
}
|
||||
function Wait-Ready($url, $seconds) {
|
||||
$deadline = (Get-Date).AddSeconds($seconds)
|
||||
while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 }
|
||||
return $false
|
||||
}
|
||||
function Stop-Both {
|
||||
foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') {
|
||||
if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') }
|
||||
Say "$s stopped"
|
||||
}
|
||||
$deadline = (Get-Date).AddSeconds(30)
|
||||
while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
|
||||
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running after stop" }
|
||||
Say "no Host.exe process remains"
|
||||
}
|
||||
function Start-Both {
|
||||
Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'
|
||||
$ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"
|
||||
if (-not $ok) { throw "central did not become ready" }
|
||||
Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'
|
||||
$ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"
|
||||
if (-not $ok) { throw "site did not become ready" }
|
||||
}
|
||||
|
||||
Say "ScadaBridge#143 deploy $fromSha -> $sha on $env:COMPUTERNAME"
|
||||
|
||||
# ---- preflight gates (nothing changed yet) ----------------------------------------------------
|
||||
if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" }
|
||||
if ((Get-FileHash "$stageDir\migrate-64-to-65.idempotent.sql" -Algorithm SHA256).Hash -ne $sqlSha256) { throw "sql hash mismatch" }
|
||||
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
|
||||
Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage
|
||||
$stagedVer = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
|
||||
$stagedCli = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion
|
||||
if ($stagedVer -notlike "*+$sha*" -or $stagedCli -notlike "*+$sha*") { throw "staged exe is $stagedVer / $stagedCli" }
|
||||
$liveVer = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
|
||||
if ($liveVer -notlike "*+$fromSha*") { throw "live exe is $liveVer, not $fromSha" }
|
||||
Say "staged $stagedVer; live $liveVer"
|
||||
Say ("staged app files {0}, cli files {1}" -f (Get-ChildItem "$stage\app" -Recurse -File).Count, (Get-ChildItem "$stage\cli" -Recurse -File).Count)
|
||||
|
||||
$envBefore = @((Get-ItemProperty $centralKey).Environment)
|
||||
$retired = @($envBefore | Where-Object { Is-Retired $_ })
|
||||
Say ("central env entries {0}; retired keys present: {1}" -f $envBefore.Count, (($retired | % { $_.Substring(0,$_.IndexOf('=')) }) -join ','))
|
||||
if (-not ($envBefore -contains 'AuditLog__HashChain__Enabled=true')) { throw "AuditLog__HashChain__Enabled=true not found in the Central env" }
|
||||
|
||||
$cs = ($envBefore | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=',''
|
||||
$sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs
|
||||
$db = $sb['Database']; if (-not $db) { $db = $sb.InitialCatalog }
|
||||
if ($db -ne 'ScadaBridge') { throw "unexpected catalog" }
|
||||
$env:SQLCMDPASSWORD = $sb.Password
|
||||
function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed: $q" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() }
|
||||
$chainQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',COL_LENGTH('dbo.AuditLog','ChainSeq'),'|',COL_LENGTH('dbo.AuditLog','RowHash'),'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainHead','U') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainTombstone','U') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID('dbo.AuditLog') AND name IN ('IX_AuditLog_ChainSeq','IX_AuditLog_Unchained')),'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog))"
|
||||
$pre = Sql-Scalar $chainQ
|
||||
Say "schema before (migrations|ChainSeq|RowHash|Head|Tomb|chainIdx|chainer|purger|writer|rows): $pre"
|
||||
if (-not $pre.StartsWith('64|')) { throw "expected 64 migrations: $pre" }
|
||||
|
||||
# ---- backups (services still running) ---------------------------------------------------------
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
|
||||
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" }
|
||||
Say "reg export $s -> $stageDir\_envbak-$s-$ts.reg"
|
||||
}
|
||||
New-Item -ItemType Directory -Force -Path $bakDir | Out-Null
|
||||
$bak = "$bakDir\ScadaBridge_pre-$($sha)_$ts.bak"
|
||||
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "backup failed" }
|
||||
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" }
|
||||
Say "db backup $bak ($((Get-Item $bak).Length) bytes, VERIFYONLY ok)"
|
||||
$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts"
|
||||
foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) }
|
||||
|
||||
# ---- outage window -------------------------------------------------------------------------
|
||||
$swapped = $false; $migrated = $false; $envChanged = $false
|
||||
try {
|
||||
Stop-Both
|
||||
|
||||
$migSql = "$stageDir\_migrate-xact-$ts.sql"
|
||||
# one UTF-8 file WITHOUT any BOM (a BOM after a prepended header broke sqlcmd in #126)
|
||||
$body = [IO.File]::ReadAllText("$stageDir\migrate-64-to-65.idempotent.sql", [Text.Encoding]::UTF8).TrimStart([char]0xFEFF)
|
||||
[IO.File]::WriteAllText($migSql, "SET XACT_ABORT ON;`r`nGO`r`n" + $body, (New-Object System.Text.UTF8Encoding $false))
|
||||
Say "migrating 64 -> 65"
|
||||
$migrated = $true
|
||||
$out = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -I -b -i $migSql 2>&1
|
||||
$out | ForEach-Object { Say " sql: $_" }
|
||||
if ($LASTEXITCODE -ne 0) { throw "migration failed (exit $LASTEXITCODE)" }
|
||||
$post = Sql-Scalar $chainQ
|
||||
Say "schema after: $post"
|
||||
if (-not $post.StartsWith('65|')) { throw "expected 65 migrations after migrate: $post" }
|
||||
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
|
||||
& sc.exe failureflag $s 1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failureflag $s failed" }
|
||||
}
|
||||
Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"; $swapped = $true
|
||||
Say "renamed app/cli -> *.bak.$ts"
|
||||
Copy-Item "$stage\app" "$root\app" -Recurse
|
||||
Copy-Item "$stage\cli" "$root\cli" -Recurse
|
||||
foreach ($f in $settings) {
|
||||
Copy-Item "$keep\$f" "$root\app\$f" -Force
|
||||
if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings restore mismatch: $f" }
|
||||
Say "restored $f (sha256 match)"
|
||||
}
|
||||
Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion)
|
||||
|
||||
$keepEnv = @($envBefore | Where-Object { -not (Is-Retired $_) })
|
||||
Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString; $envChanged = $true
|
||||
$envAfter = @((Get-ItemProperty $centralKey).Environment)
|
||||
foreach ($e in $keepEnv) { if ($envAfter -notcontains $e) { throw "an existing env entry changed" } }
|
||||
if (@($envAfter | Where-Object { Is-Retired $_ }).Count -ne 0) { throw "retired env key still present" }
|
||||
Say ("central env entries {0} -> {1}" -f $envBefore.Count, $envAfter.Count)
|
||||
Digest $envAfter | ForEach-Object { Say " env $_" }
|
||||
|
||||
Start-Both
|
||||
}
|
||||
catch {
|
||||
Say "ERROR: $($_.Exception.Message)"
|
||||
Say "AUTO-ROLLBACK (migrated=$migrated swapped=$swapped envChanged=$envChanged)"
|
||||
try {
|
||||
Stop-Both
|
||||
if ($migrated) {
|
||||
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "ALTER DATABASE [$db] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [$db] FROM DISK = N'$bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [$db] SET MULTI_USER;" | ForEach-Object { Say " restore: $_" }
|
||||
if ($LASTEXITCODE -ne 0) { throw "DB restore failed" }
|
||||
Say "DB restored from $bak"
|
||||
}
|
||||
if ($swapped) {
|
||||
if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" }
|
||||
if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" }
|
||||
Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli'
|
||||
Say "restored app/cli from *.bak.$ts"
|
||||
}
|
||||
if ($envChanged) { Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$envBefore) -Type MultiString; Say "central env restored" }
|
||||
Start-Both
|
||||
Say ("rolled back; live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion)
|
||||
} catch { Say "AUTO-ROLLBACK FAILED: $($_.Exception.Message) - manual action needed" }
|
||||
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ---- post checks (verify of this deploy) ------------------------------------------------------
|
||||
Start-Sleep 20
|
||||
$q2 = Sql-Scalar "SELECT CONCAT(COUNT_BIG(*),'|',CONVERT(varchar(33),MAX(OccurredAtUtc),126)) FROM dbo.AuditLog"
|
||||
Say "auditlog rows|newest: $q2"
|
||||
$logDir = "$root\central\logs"
|
||||
if (Test-Path $logDir) {
|
||||
$f = Get-ChildItem $logDir -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1
|
||||
$hits = @(Select-String -Path $f.FullName -Pattern 'no longer supported' -SimpleMatch)
|
||||
Say ("central log {0}: retired-key warnings since start = {1}" -f $f.Name, $hits.Count)
|
||||
}
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { Say ("$s " + (Get-Service $s).Status + " / " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim()) }
|
||||
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
|
||||
Say "DEPLOY DONE ts=$ts backup=$bak"
|
||||
exit 0
|
||||
@@ -0,0 +1,27 @@
|
||||
# #116 QA site-pair drill helper. Rendered per call: ACTION. Service ScadaBridge-Site only.
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$Action = '__ACTION__'
|
||||
$svc = 'ScadaBridge-Site'
|
||||
function Say($m) { "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m }
|
||||
function Rec { Say ("qfailure: " + ((& sc.exe qfailure $svc | Out-String) -replace '\s+',' ').Trim()); Say ("qfailureflag: " + ((& sc.exe qfailureflag $svc | Out-String) -replace '\s+',' ').Trim()) }
|
||||
$pidNow = (Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId
|
||||
switch ($Action) {
|
||||
'status' { Say "$svc state $((Get-Service $svc).Status) pid $pidNow"; Rec }
|
||||
'kill' {
|
||||
if (-not $pidNow -or $pidNow -eq 0) { throw "no running pid for $svc" }
|
||||
$p = Get-Process -Id $pidNow; if ($p.ProcessName -ne 'ZB.MOM.WW.ScadaBridge.Host') { throw "pid $pidNow is $($p.ProcessName)" }
|
||||
Say "KILL $svc pid $pidNow"; Stop-Process -Id $pidNow -Force; Say "killed" }
|
||||
'disable' {
|
||||
& sc.exe failure $svc reset= 86400 actions= none/0/none/0/none/0 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failure disable" }
|
||||
$q = (& sc.exe qfailure $svc | Out-String); if ($q -match 'RESTART') { throw "restart action still present" }; Rec }
|
||||
'kill-held' {
|
||||
$q = (& sc.exe qfailure $svc | Out-String); if ($q -match 'RESTART') { throw "refusing: restart action present" }
|
||||
$p = Get-Process -Id $pidNow; if ($p.ProcessName -ne 'ZB.MOM.WW.ScadaBridge.Host') { throw "pid $pidNow is $($p.ProcessName)" }
|
||||
Say "KILL-HELD $svc pid $pidNow"; Stop-Process -Id $pidNow -Force; Say "killed (no auto-restart)" }
|
||||
'restore' {
|
||||
& sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "sc failure restore" }
|
||||
& sc.exe failureflag $svc 1 | Out-Null; Rec }
|
||||
'start' { Say "starting $svc"; Start-Service $svc; (Get-Service $svc).WaitForStatus('Running','00:01:00'); Say "running pid $((Get-CimInstance Win32_Service -Filter "Name='$svc'").ProcessId)" }
|
||||
default { throw "bad action" }
|
||||
}
|
||||
exit 0
|
||||
@@ -0,0 +1,86 @@
|
||||
BEGIN TRANSACTION;
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_ChainSeq' AND object_id = OBJECT_ID('dbo.AuditLog'))
|
||||
EXEC(N'DROP INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog;');
|
||||
IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_Unchained' AND object_id = OBJECT_ID('dbo.AuditLog'))
|
||||
EXEC(N'DROP INDEX IX_AuditLog_Unchained ON dbo.AuditLog;');
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NOT NULL
|
||||
BEGIN
|
||||
IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL
|
||||
EXEC(N'REVOKE UPDATE (ChainSeq, RowHash) ON dbo.AuditLog FROM scadabridge_audit_chainer;');
|
||||
EXEC(N'REVOKE SELECT ON dbo.AuditLog FROM scadabridge_audit_chainer;');
|
||||
|
||||
DECLARE @dropMembers nvarchar(max) = N'';
|
||||
SELECT @dropMembers = @dropMembers + N'ALTER ROLE scadabridge_audit_chainer DROP MEMBER ' + QUOTENAME(m.name) + N';' + CHAR(10)
|
||||
FROM sys.database_role_members rm
|
||||
JOIN sys.database_principals r ON r.principal_id = rm.role_principal_id
|
||||
JOIN sys.database_principals m ON m.principal_id = rm.member_principal_id
|
||||
WHERE r.name = N'scadabridge_audit_chainer';
|
||||
EXEC sp_executesql @dropMembers;
|
||||
|
||||
EXEC(N'DROP ROLE scadabridge_audit_chainer;');
|
||||
END
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF OBJECT_ID('dbo.AuditLogChainHead', 'U') IS NOT NULL
|
||||
EXEC(N'DROP TABLE dbo.AuditLogChainHead;');
|
||||
IF OBJECT_ID('dbo.AuditLogChainTombstone', 'U') IS NOT NULL
|
||||
EXEC(N'DROP TABLE dbo.AuditLogChainTombstone;');
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL
|
||||
EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN ChainSeq;');
|
||||
IF COL_LENGTH('dbo.AuditLog', 'RowHash') IS NOT NULL
|
||||
EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN RowHash;');
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
|
||||
IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NOT NULL
|
||||
BEGIN
|
||||
EXEC(N'REVOKE ALTER ANY DATASPACE FROM scadabridge_audit_purger;');
|
||||
EXEC(N'REVOKE UPDATE (SourceNode) ON dbo.AuditLog FROM scadabridge_audit_purger;');
|
||||
END
|
||||
END;
|
||||
|
||||
IF NOT EXISTS (
|
||||
SELECT * FROM [__EFMigrationsHistory]
|
||||
WHERE [MigrationId] = N'20261115221558_RemoveAuditLogHashChain'
|
||||
)
|
||||
BEGIN
|
||||
INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion])
|
||||
VALUES (N'20261115221558_RemoveAuditLogHashChain', N'10.0.13');
|
||||
END;
|
||||
|
||||
COMMIT;
|
||||
GO
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
# ScadaBridge QA (vq03/vq04) 8e32f353 -> 061432c8 : ScadaBridge#143. Rendered per call: ACTION / TS.
|
||||
# prep | backup (vq03) | stop-site | stop-central | migrate (vq03) | swap | start-central | start-site | verify | rollback-swap
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$ProgressPreference = 'SilentlyContinue'
|
||||
$Action = '__ACTION__'
|
||||
$ts = '__TS__'
|
||||
$sha = '061432c8'
|
||||
$fromSha = '8e32f353'
|
||||
$zipSha256 = 'B6E85ECC24186AF8B658FA6F1E38F080588EA5F78DD856A4356E696897FF4095'
|
||||
$sqlSha256 = 'A16A150DE1E1F6C4CB9153E1C53BED95A029D656F18C3DC3B61EC6575198F7A7'
|
||||
$stageDir = "E:\psrem-stage\sb-$sha"
|
||||
$stage = "$stageDir\unpacked"
|
||||
$root = 'E:\ApiInstall\ScadaBridge'
|
||||
$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt"
|
||||
$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json'
|
||||
$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central'
|
||||
$bak = "F:\MSSQL\BACKUP\ScadaBridge_pre-$($sha)_$ts.bak"
|
||||
function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m; $line; Add-Content -Path $log -Value $line }
|
||||
function Is-Retired($e) { $e -like 'AuditLog__HashChain__*' -or $e -like 'AuditLog__MaintenanceConnectionString=*' }
|
||||
function Digest($entries) {
|
||||
$h = [System.Security.Cryptography.SHA256]::Create(); $o = @()
|
||||
foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += '<unparsable>'; continue }
|
||||
$v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') })
|
||||
$o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) }
|
||||
$h.Dispose(); $o | Sort-Object
|
||||
}
|
||||
function Wait-Ready($url, $seconds) {
|
||||
$deadline = (Get-Date).AddSeconds($seconds)
|
||||
while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 }
|
||||
return $false
|
||||
}
|
||||
function Stop-Svc($s) {
|
||||
if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') }
|
||||
Say "$s stopped"
|
||||
}
|
||||
function Sql-Init {
|
||||
$envC = @((Get-ItemProperty $centralKey).Environment)
|
||||
$cs = ($envC | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=',''
|
||||
$script:sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs
|
||||
$script:db = $sb['Database']; if (-not $db) { $script:db = $sb.InitialCatalog }
|
||||
if ($db -ne 'ScadaBridge') { throw "unexpected catalog" }
|
||||
$env:SQLCMDPASSWORD = $sb.Password
|
||||
}
|
||||
function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() }
|
||||
$chainQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',COL_LENGTH('dbo.AuditLog','ChainSeq'),'|',COL_LENGTH('dbo.AuditLog','RowHash'),'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainHead','U') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN OBJECT_ID('dbo.AuditLogChainTombstone','U') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT(*) FROM sys.indexes WHERE object_id=OBJECT_ID('dbo.AuditLog') AND name IN ('IX_AuditLog_ChainSeq','IX_AuditLog_Unchained')),'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL THEN 0 ELSE 1 END,'|',CASE WHEN DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL THEN 0 ELSE 1 END,'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog),'|',CONVERT(varchar(33),(SELECT MAX(OccurredAtUtc) FROM dbo.AuditLog),126))"
|
||||
|
||||
switch ($Action) {
|
||||
'prep' {
|
||||
Say "PREP $fromSha -> $sha"
|
||||
if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" }
|
||||
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
|
||||
Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage
|
||||
$sv = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
|
||||
$sc = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion
|
||||
if ($sv -notlike "*+$sha*" -or $sc -notlike "*+$sha*") { throw "staged exe $sv / $sc" }
|
||||
$lv = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
|
||||
if ($lv -notlike "*+$fromSha*") { throw "live exe $lv" }
|
||||
Say "staged $sv; live $lv; app files $((Get-ChildItem "$stage\app" -Recurse -File).Count), cli $((Get-ChildItem "$stage\cli" -Recurse -File).Count)"
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
|
||||
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" }
|
||||
Say "reg export $s"
|
||||
$e = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment)
|
||||
Say ("$s env entries {0}" -f $e.Count); Digest $e | % { Say " env $s $_" }
|
||||
Say ("recovery ${s}: " + ((& sc.exe qfailure $s | Out-String) -replace '\s+',' ').Trim())
|
||||
Say ("recovery ${s}: " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim())
|
||||
}
|
||||
$envC = @((Get-ItemProperty $centralKey).Environment)
|
||||
if (-not ($envC -contains 'AuditLog__HashChain__Enabled=true')) { throw "AuditLog__HashChain__Enabled=true not in Central env" }
|
||||
Say ("retired keys: " + ((@($envC | ? { Is-Retired $_ }) | % { $_.Substring(0,$_.IndexOf('=')) }) -join ','))
|
||||
$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts"
|
||||
foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) }
|
||||
}
|
||||
'backup' {
|
||||
Sql-Init
|
||||
if ((Get-FileHash "$stageDir\migrate-64-to-65.idempotent.sql" -Algorithm SHA256).Hash -ne $sqlSha256) { throw "sql hash mismatch" }
|
||||
$pre = Sql-Scalar $chainQ; Say "schema before: $pre"
|
||||
if (-not $pre.StartsWith('64|')) { throw "expected 64 migrations" }
|
||||
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "backup failed" }
|
||||
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" }
|
||||
Say "backup $bak VERIFYONLY ok"
|
||||
}
|
||||
'stop-site' { Stop-Svc 'ScadaBridge-Site' }
|
||||
'stop-central' {
|
||||
Stop-Svc 'ScadaBridge-Central'
|
||||
$deadline = (Get-Date).AddSeconds(30)
|
||||
while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
|
||||
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running" }
|
||||
Say "no Host.exe remains"
|
||||
}
|
||||
'migrate' {
|
||||
Sql-Init
|
||||
foreach ($h in 'wonder-app-vq03','wonder-app-vq04') { foreach ($p in 8085,8084) { try { Invoke-WebRequest -UseBasicParsing "http://$($h):$p/health/ready" -TimeoutSec 3 | Out-Null; throw "$h`:$p still answering - not all services stopped" } catch [System.Net.WebException] {} } }
|
||||
$migSql = "$stageDir\_migrate-xact-$ts.sql"
|
||||
$body = [IO.File]::ReadAllText("$stageDir\migrate-64-to-65.idempotent.sql", [Text.Encoding]::UTF8).TrimStart([char]0xFEFF)
|
||||
[IO.File]::WriteAllText($migSql, "SET XACT_ABORT ON;`r`nGO`r`n" + $body, (New-Object System.Text.UTF8Encoding $false))
|
||||
Say "migrating 64 -> 65"
|
||||
$out = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -I -b -i $migSql 2>&1
|
||||
$rc = $LASTEXITCODE
|
||||
$out | % { Say " sql: $_" }
|
||||
if ($rc -ne 0) { throw "migration failed (exit $rc)" }
|
||||
$post = Sql-Scalar $chainQ; Say "schema after: $post"
|
||||
if (-not $post.StartsWith('65|')) { throw "expected 65 migrations" }
|
||||
}
|
||||
'swap' {
|
||||
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe running" }
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { & sc.exe failureflag $s 1 | Out-Null; if ($LASTEXITCODE -ne 0) { throw "failureflag $s" } }
|
||||
Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"
|
||||
Say "renamed app/cli -> *.bak.$ts"
|
||||
Copy-Item "$stage\app" "$root\app" -Recurse
|
||||
Copy-Item "$stage\cli" "$root\cli" -Recurse
|
||||
$keep = "$root\_appsettings-keep-$ts"
|
||||
foreach ($f in $settings) {
|
||||
Copy-Item "$keep\$f" "$root\app\$f" -Force
|
||||
if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings mismatch $f" }
|
||||
Say "restored $f (sha256 match)"
|
||||
}
|
||||
Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion)
|
||||
$before = @((Get-ItemProperty $centralKey).Environment)
|
||||
$keepEnv = @($before | Where-Object { -not (Is-Retired $_) })
|
||||
Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString
|
||||
$after = @((Get-ItemProperty $centralKey).Environment)
|
||||
foreach ($e in $keepEnv) { if ($after -notcontains $e) { throw "env entry changed" } }
|
||||
if (@($after | ? { Is-Retired $_ }).Count -ne 0) { throw "retired key still present" }
|
||||
Say ("central env entries {0} -> {1}" -f $before.Count, $after.Count); Digest $after | % { Say " env ScadaBridge-Central $_" }
|
||||
}
|
||||
'start-central' { Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'; $ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"; if (-not $ok) { throw "central not ready" } }
|
||||
'start-site' { Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'; $ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"; if (-not $ok) { throw "site not ready" } }
|
||||
'verify' {
|
||||
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
|
||||
Say ("$s " + (Get-Service $s).Status)
|
||||
Say ("recovery ${s}: " + ((& sc.exe qfailure $s | Out-String) -replace '\s+',' ').Trim())
|
||||
Say ("recovery ${s}: " + ((& sc.exe qfailureflag $s | Out-String) -replace '\s+',' ').Trim())
|
||||
Digest @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment) | % { Say " env $s $_" }
|
||||
}
|
||||
$logDir = "$root\central\logs"
|
||||
$f = Get-ChildItem $logDir -File | Sort-Object LastWriteTime -Descending | Select-Object -First 1
|
||||
Say ("central log {0}: retired-key warnings = {1}" -f $f.Name, @(Select-String -Path $f.FullName -Pattern 'no longer supported' -SimpleMatch).Count)
|
||||
if ($env:COMPUTERNAME -eq 'WONDER-APP-VQ03') { Sql-Init; Say ("schema now: " + (Sql-Scalar $chainQ)) }
|
||||
}
|
||||
'rollback-swap' {
|
||||
foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') { Stop-Svc $s }
|
||||
if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" }
|
||||
if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" }
|
||||
Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli'
|
||||
& reg.exe import "$stageDir\_envbak-ScadaBridge-Central-$ts.reg" | Out-Null
|
||||
Say "rolled back binaries + Central env (services left STOPPED; restore DB next)"
|
||||
}
|
||||
default { throw "bad action $Action" }
|
||||
}
|
||||
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
|
||||
Say "$Action OK"
|
||||
exit 0
|
||||
@@ -0,0 +1,68 @@
|
||||
# 2026-11-15: DEV (wonder-app-vd03) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143)
|
||||
|
||||
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". IT security asked for the AuditLog hash chain to be removed. The pack is `docs/deployment/deploy-prep/2026-11-15-143/`.
|
||||
|
||||
**Build:** `main` @ `061432c8`, ProductVersion `1.0.0+061432c8c4299505a45d4cf396c3d62139865080`.
|
||||
- Host and CLI were published self-contained win-x64 from a clean detached worktree: 1056 app files and 224 cli files.
|
||||
- The zip is 130 414 127 B, SHA-256 `b6e85ecc24186af8b658fa6f1e38f080588ea5f78dd856a4356e696897ff4095`.
|
||||
- Pins: LocalDb 0.3.8, Theme 1.0.1.
|
||||
|
||||
**Delta:** 25 commits. See the pack: the hash-chain, head-export and two-login removal (`3dea146c`), #141, #105, and the UI-consistency / Theme 1.0.x work. There is one migration, `20261115221558_RemoveAuditLogHashChain`. The script `migrate-64-to-65.idempotent.sql` has SHA-256 `a16a150d…98f7a7`, is identical to `docs/plans/sql/RemoveAuditLogHashChain.sql` once comments are stripped, and was run with `SET XACT_ABORT ON` and `sqlcmd -I -b`.
|
||||
|
||||
**Transport:** one deploy script over `winrm` (deploy scope, owner-authorized), `dev-deploy.ps1` in `E:\ApiInstall\_stage-sb-061432c8\`. Staging went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191258.txt`.
|
||||
|
||||
**Not touched:** QA (see its own record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector. Nothing was deployed to any instance.
|
||||
|
||||
## Preflight (in the script, before anything changed)
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Live build | `8e32f353` |
|
||||
| Staged build | `061432c8` Host + CLI |
|
||||
| Schema | 64 migrations; `ChainSeq` (8) / `RowHash` (32) present, both chain tables, 2 chain indexes, roles `_chainer`, `_purger`, `_writer`; AuditLog 2 853 rows |
|
||||
| Central env | 13 entries; retired key present: `AuditLog__HashChain__Enabled` (= `true`) |
|
||||
| appsettings | `A6B6CF05…`, `3382DD83…`, `3E7AD9A4…` (unchanged since 11-13) |
|
||||
| Health / baseline | central and site ready/active 200; Z28062 270 attrs, Good 144, 8 alarms |
|
||||
|
||||
## Window
|
||||
|
||||
| Step | Local (EST) |
|
||||
|---|---|
|
||||
| `reg export` of both service keys; backup `E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak` (131 757 056 B, `COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); appsettings kept in `_appsettings-keep-20261115T191258` | 19:13:11 → 19:13:14 |
|
||||
| Stop Site, then Central; no `Host.exe` left | 19:13:14 → 19:13:35 |
|
||||
| Migrate 64 → 65 | 19:13:35 (< 1 s) |
|
||||
| `failureflag 1` re-asserted; `app`/`cli` → `*.bak.20261115T191258`; new build in; appsettings restored (hash match) | 19:13:35 → 19:13:41 |
|
||||
| Central env: `AuditLog__HashChain__Enabled` removed, 13 → 12 entries; the other 12 byte-identical (digest-checked) | 19:13:41 |
|
||||
| Central ready, then Site ready | 19:13:53 / 19:14:02 |
|
||||
|
||||
**Downtime:** the site data path was down for about 48 s (19:13:14 → 19:14:02) and the central API for about 29 s (19:13:25 → 19:13:54).
|
||||
|
||||
## Gates (all pass)
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Build | Host + CLI `1.0.0+061432c8…` |
|
||||
| Schema | **65** migrations; `ChainSeq`, `RowHash`, `AuditLogChainHead`, `AuditLogChainTombstone`, `IX_AuditLog_ChainSeq`, `IX_AuditLog_Unchained` and `scadabridge_audit_chainer` all **absent**; `scadabridge_audit_purger` + `_writer` present; 2 853 rows kept |
|
||||
| Health | ready/active 200/200 (central `:8085`, site `:8084`). Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone). Site checks: `akka-cluster`, `localdb` Healthy |
|
||||
| Retired-key warning | 0 in `scadabridge-central-20261115.log`, because the env is clean |
|
||||
| AuditLog | new rows after the start: 2 `InboundAuthFailure` from `central-a` (the bogus-key probes) |
|
||||
| Inbound API | bogus `X-API-Key` → **401** ×2 |
|
||||
| MxGateway / instances | `MxGateway` Connected, 810/810 resolved, good 398 / bad 412 (= baseline); Z28062 Good 144 / 270, 8 alarms (= baseline); 5 deployed instances |
|
||||
| CLI | `audit` group: query, export, tree, backfill-source-node, archive; **no** `verify-chain` / `verify-heads` |
|
||||
| OTLP | box collector `logs.jsonl` carries `service.version 1.0.0+061432c8…` (79 records at check time) |
|
||||
| Overview `:5320` | 6/6 Up; ScadaBridge central Active/Up, leader chip, 4/4 checks |
|
||||
| Recovery | both services `FAILURE_ACTIONS_ON_NONCRASH_FAILURES: TRUE` |
|
||||
| Pending updates | 1 stale instance: Z28061, expected. Its machine moved to QA, and it was left on the old #52 body on purpose (#142 closed) |
|
||||
|
||||
The env block still has `ScadaBridge__Database__MachineDataDb`, which has been unused since its removal on 2026-10-22. It is harmless and was not touched.
|
||||
|
||||
## Rollback
|
||||
|
||||
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore:
|
||||
1. Stop Site, then Central.
|
||||
2. `RESTORE DATABASE [ScadaBridge] FROM DISK = N'E:\ApiInstall\ScadaBridge-backups\ScadaBridge_pre-061432c8_20261115T191258.bak' WITH REPLACE, CHECKSUM` (single-user first).
|
||||
3. `app`/`cli` ← `*.bak.20261115T191258`.
|
||||
4. `reg import E:\ApiInstall\_stage-sb-061432c8\_envbak-ScadaBridge-Central-20261115T191258.reg`, which puts `AuditLog__HashChain__Enabled=true` back.
|
||||
5. Start Central, then Site.
|
||||
|
||||
The restore loses every DB write since 19:13. The `*.bak.20261115T191258` and `_appsettings-keep-20261115T191258` folders can go once the build has soaked.
|
||||
@@ -0,0 +1,116 @@
|
||||
# 2026-11-15: QA (wonder-app-vq03 + vq04) `8e32f353` → `061432c8`, migration 64 → 65, audit hash chain removed (ScadaBridge#143), #116 site-pair drills
|
||||
|
||||
**Owner go (2026-11-15):** "yes 143, with the 116 dependent on idle". The pack is `docs/deployment/deploy-prep/2026-11-15-143/`. DEV went first (`2026-11-15-wonder-app-vd03-061432c8.md`, all gates passed).
|
||||
|
||||
**Build:** the same zip as DEV, `061432c8`, SHA-256 `b6e85ecc…ff4095`. It was pushed to `E:\psrem-stage\sb-061432c8\` on both boxes and its hash re-verified on each box: 1056 app files, 224 cli files.
|
||||
|
||||
**Transport:** one rendered script per step over `winrm` (deploy scope, owner-authorized): `prep`, `backup`, `stop-site`, `stop-central`, `migrate`, `swap`, `start-central`, `start-site`, `verify`. Staging and logs went over SMB, and the gates were checked over HTTP and with the CLI from the Mac. The backup and migration ran on vq03 against wonder-sql-vq03 (vq04 has no `sqlcmd`), using the Central service's own connection string with the password in `SQLCMDPASSWORD` only. The on-box notes are `E:\ApiInstall\ScadaBridge\_DEPLOY-NOTES-061432c8-20261115T191851.txt`.
|
||||
|
||||
**Not touched:** DEV (separate record), OtOpcUa, mxaccessgw, HistorianGateway, Overview, the OTel collector, `ScadaBridge-LDAP`. No instance was deployed to.
|
||||
|
||||
## Preflight
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Live / staged | `8e32f353` / `061432c8` on both boxes |
|
||||
| Schema (wonder-sql-vq03) | 64 migrations; chain columns, tables, indexes and `_chainer` present; AuditLog 3 606 rows |
|
||||
| Service env | Central 16 entries, Site 8 entries, **digest-identical on both boxes**; retired key `AuditLog__HashChain__Enabled` (`true`) on both Centrals |
|
||||
| appsettings | `C0C653D6…` (both); Central vq03 `EF01B129…`, vq04 `F333D48F…`; Site vq03 `D0E202F9…`, vq04 `E9E6336F…`. These equal the 11-13 post-deploy hashes, and the per-node JWT/idle drift is kept |
|
||||
| Recovery | all four: restart 5 s / 5 s / 10 s, reset 86 400, non-crash flag TRUE |
|
||||
| Active | central vq03, site vq04 (since the #116 central drills earlier today) |
|
||||
| Baseline | Z28061–Z28069 244/244 Good, 10 alarms each; WarmCanary 2/2 Good; MxGateway 2196/2196 good; LocalDb sync connected |
|
||||
|
||||
## Window
|
||||
|
||||
| Step | Local (EST) |
|
||||
|---|---|
|
||||
| Backup `F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak` (`COPY_ONLY, CHECKSUM, COMPRESSION`, VERIFYONLY ok); `reg export` + appsettings kept on both boxes in the prep step | 19:19:00 → 19:19:48 |
|
||||
| Stop Site vq03 (standby) | 19:20:06 → 19:20:17 |
|
||||
| Stop Site vq04 (active, **site data path down**) | 19:20:34 → 19:20:45 |
|
||||
| Stop Central vq04 (standby) | 19:21:02 → 19:21:12 |
|
||||
| Stop Central vq03 (active, **central API down**); no `Host.exe` on either box | 19:21:23 → 19:21:33 |
|
||||
| Migrate 64 → 65 (it refuses to run while any ScadaBridge port still answers) | 19:21:54 (< 1 s) |
|
||||
| Swap on both boxes in parallel: `failureflag 1`, `app`/`cli` → `*.bak.20261115T191851`, new build, appsettings restored (hash match), Central env 16 → 15 | 19:22:16 → 19:22:31 |
|
||||
| Start Centrals (vq03, vq04 3 s later) → ready | 19:23:00 / 19:23:05 |
|
||||
| Start Sites (vq03, vq04 3 s later) → ready | 19:23:25 / 19:23:30 |
|
||||
|
||||
**Downtime:** the site data path was down for about 2 min 40 s (19:20:35 → 19:23:25) and the central API for about 1 min 37 s (19:21:23 → 19:23:00). Most of the gaps between steps are the ~15 s WinRM session set-up per call.
|
||||
|
||||
## Gates (all pass)
|
||||
|
||||
| Gate | Result |
|
||||
|---|---|
|
||||
| Build | Host + CLI `1.0.0+061432c8…` on both boxes |
|
||||
| Schema | **65** migrations; `ChainSeq`, `RowHash`, both chain tables, both chain indexes and `scadabridge_audit_chainer` **absent**; `_purger` + `_writer` present; 3 606 rows kept |
|
||||
| Health | ready 200 ×4; active: central vq03 200 / vq04 503, site vq03 200 / vq04 503. Central checks: `database`, `akka-cluster`, `required-singletons`, `bundle-signing-policy` Healthy (`audit-chain-head-export` gone) |
|
||||
| Env | Central env = the pre-window entries minus `AuditLog__HashChain__Enabled`, byte-identical otherwise, identical on both boxes; Site env unchanged |
|
||||
| Retired-key warning | 0 on both Centrals |
|
||||
| Cluster / LocalDb | central Primary vq03 / Standby vq04, both online; site LocalDb `replicationConnected: true`, backlog 0 on both nodes |
|
||||
| Inbound API | bogus key → **401** on vq03 and vq04 (through the standby relay); 2 new `InboundAuthFailure` rows from `central-a` |
|
||||
| API keys | `MESTestKey`, `MES_Key`, `delmia-notifier-qa` |
|
||||
| Data | MxGateway 2196/2196 good; WarmCanary connected, 2/2 good; Z28061–Z28069 244/244 Good, 10 alarms each (= baseline); 10 deployed instances |
|
||||
| CLI | `audit` group has no `verify-chain` / `verify-heads` |
|
||||
| OTLP | box collectors carry `service.version 1.0.0+061432c8…` (vq03 95, vq04 63 log records at check time) |
|
||||
| Overview `:5320` | 12/12 Up on vq03 and vq04; ScadaBridge 4 instances / 2 clusters, central leader vq03 |
|
||||
| Recovery | unchanged on all four services |
|
||||
| Pending updates | 1 stale: **WarmCanary** (16). Its OPC UA connection config was edited after its 2026-11-13 deploy (#138 work). This deploy did not cause it, and it was left alone |
|
||||
|
||||
**Purge/partition sanity:** skipped. There is no read-only way to exercise a purge or switch-out, and nothing is due: archive and retention are off.
|
||||
|
||||
## #116 site-pair drills (owner condition: only if idle)
|
||||
|
||||
**Idle check:** two snapshots of all nine CvdReactors (Z28061–Z28069), at 00:27:18Z and 00:28:38Z.
|
||||
- Every side showed `CycleRunning` False (Good) and `MachineCycleID` 0.
|
||||
- FurnaceTemp was 23–25 °C, and no `RecipeDownloadFlag` was True.
|
||||
- Between the snapshots the only changes were ±1 argon purge flow (Z28061, Z28067) and ±1 °C (Z28065).
|
||||
|
||||
So the drills ran. ZIPSEN is disabled and hosts no attributes.
|
||||
|
||||
**Kill drill.** The active Site, vq03 (PID 8896, picked through the service), was killed at 00:29:06.0Z:
|
||||
|
||||
| Event | UTC | After kill |
|
||||
|---|---|---|
|
||||
| vq03 Site `/health` stops answering | 00:29:06.4 | +0.4 s |
|
||||
| vq04 `New incarnation of existing member … is trying to join`; `[leader-auto-down] … already "Down"; not scheduling a Down`; vq04 gains leadership | 00:29:13.05 | +7.0 s |
|
||||
| vq04 Site active 200 | 00:29:13.7 | **+7.7 s** |
|
||||
| vq03 back as standby (ready 200 / active 503) | 00:29:24.3 | +18.3 s |
|
||||
|
||||
As in the central kill drill, the SCM restarts the killed process after 5 s, before the 15 s downing timer runs out, so Akka downs the old incarnation itself. After the drill all 10 instances were 2198/2198 Good on vq04, and LocalDb was connected.
|
||||
|
||||
**Stop-and-hold drill** on the new active Site, vq04 (PID 18956), after about 1 min 20 s stable:
|
||||
1. Recovery config recorded: RESTART 5000/5000/10000, reset 86400, non-crash flag TRUE.
|
||||
2. Set to no actions. The kill script refuses to run while any RESTART action remains.
|
||||
3. Killed at 00:30:32.7Z.
|
||||
4. Held down for 113 s.
|
||||
5. Recovery restored **byte-identical** to the recorded config (diff-checked), then started.
|
||||
|
||||
| Event | UTC | After kill |
|
||||
|---|---|---|
|
||||
| Kill vq04 Site | 00:30:32.72 | 0 |
|
||||
| vq03 `[leader-auto-down] … 10.220.158.3:8082 is unreachable; it will be downed if still unreachable after "00:00:15"` | 00:30:44.86 | +12.1 s |
|
||||
| vq03 `[WRN] [leader-auto-down] DOWNING akka.tcp://scadabridge@10.220.158.3:8082: unreachable for "00:00:15" and this node (akka.tcp://scadabridge@10.220.158.2:8082) is leader` | 00:30:59.87 | **+27.2 s** |
|
||||
| vq03 Site active 200 | 00:31:00.17 | **+27.4 s** |
|
||||
| Recovery restored, `Start-Service` | 00:31:56 / 00:32:25 | — |
|
||||
| vq04 `Welcome from …10.220.158.2:8082`; standby (ready 200 / active 503) | 00:32:27.8 / 00:32:28.2 | 2.7 s after the start |
|
||||
|
||||
vq04 rejoined without the #132 shut-out path (no exit 75), because it started after it had been removed.
|
||||
|
||||
**After the drills (all pass):**
|
||||
- ready 200 ×4;
|
||||
- one active per pair: central vq03 and site vq03;
|
||||
- 10 instances at 2198/2198 Good;
|
||||
- MxGateway 2196 good, WarmCanary 2 good;
|
||||
- LocalDb connected, backlog 0, on both nodes;
|
||||
- both Site services back on RESTART 5 s / 5 s / 10 s, reset 86 400, flag TRUE;
|
||||
- Overview 12/12 Up.
|
||||
|
||||
**What this proves on QA:** a killed site node fails over in about 8 s through the SCM-restart path. A site node that stays down is downed by `leader-auto-down` after 15 s unreachable, and the pair takes over in about 27 s. Both match the central drills (`2026-11-15-qa-116-central-drill.md`).
|
||||
|
||||
## Rollback (deploy)
|
||||
|
||||
Schema 65 cannot run under `8e32f353`, so a rollback is binaries **plus** a DB restore, with the pair stopped together:
|
||||
1. On each box, run `qa.tmpl.ps1` (in the pack and in `E:\psrem-stage\sb-061432c8\`) rendered with `__ACTION__`=`rollback-swap` and `__TS__`=`20261115T191851`. It stops Site then Central, puts back `app`/`cli` ← `*.bak.20261115T191851` and re-imports the pre-window Central key (`_envbak-ScadaBridge-Central-20261115T191851.reg`). Services are left stopped.
|
||||
2. With all four stopped, from vq03: `ALTER DATABASE [ScadaBridge] SET SINGLE_USER WITH ROLLBACK IMMEDIATE; RESTORE DATABASE [ScadaBridge] FROM DISK = N'F:\MSSQL\BACKUP\ScadaBridge_pre-061432c8_20261115T191851.bak' WITH REPLACE, CHECKSUM; ALTER DATABASE [ScadaBridge] SET MULTI_USER;`. This loses every DB write since 19:19.
|
||||
3. Start both Centrals together, then both Sites.
|
||||
|
||||
The `*.bak.20261115T191851` and `_appsettings-keep-20261115T191851` folders can go once the build has soaked.
|
||||
Reference in New Issue
Block a user