From bfc369062c95a4ea0fe6f2ea6a7675b347859c82 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 19 Nov 2026 17:26:12 -0500 Subject: [PATCH] deploy(prod): stage the PROD install pack on app-vp03/04 + par-vp03/04 over SMB (nothing executed) Configs, env templates, install wrappers (dry/real, never start), SQL (65-migration schema, zbsecrets v1, _PROD role rows), preflight, seed script, patched mxgw Install script (worker name, file KEK, connstr mask). Plan: Phase 3 gate corrected. Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ --- docs/deploy/2026-11-19-prod-stage/README.md | 95 + .../common/preflight-prod.ps1 | 65 + .../common/seed-secret-prod.ps1 | 56 + .../mxgw/Install-MxGatewayService.ps1 | 213 ++ .../mxgw/Upgrade-MxGatewayService.ps1 | 869 +++++++ .../mxgw/appsettings.json.prod | 117 + .../mxgw/install-mxgw.ps1 | 99 + .../2026-11-19-prod-stage/mxgw/mxgw.env | 42 + .../otel/Install-OtelCollector.ps1 | 116 + .../otel/Test-OtelCollector.ps1 | 65 + .../2026-11-19-prod-stage/otel/config.yaml | 80 + .../sb/appsettings.Central.json.prod-vp03 | 113 + .../sb/appsettings.Central.json.prod-vp04 | 113 + .../sb/appsettings.Site.json.prod-parvp03 | 89 + .../sb/appsettings.Site.json.prod-parvp04 | 88 + .../2026-11-19-prod-stage/sb/install-sb.ps1 | 110 + .../2026-11-19-prod-stage/sb/sb-central.env | 15 + .../2026-11-19-prod-stage/sb/sb-site.env | 11 + .../sql/sb-role-mappings-prod.sql | 20 + .../sql/scadabridge-schema-2e0ce81e.sql | 2113 +++++++++++++++++ .../sql/zbsecrets-schema-v1.sql | 44 + ...026-11-19-prod-scadabridge-mxgw-install.md | 9 +- 22 files changed, 4539 insertions(+), 3 deletions(-) create mode 100644 docs/deploy/2026-11-19-prod-stage/README.md create mode 100644 docs/deploy/2026-11-19-prod-stage/common/preflight-prod.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/common/seed-secret-prod.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/mxgw/Upgrade-MxGatewayService.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/mxgw/appsettings.json.prod create mode 100644 docs/deploy/2026-11-19-prod-stage/mxgw/install-mxgw.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/mxgw/mxgw.env create mode 100644 docs/deploy/2026-11-19-prod-stage/otel/Install-OtelCollector.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/otel/Test-OtelCollector.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/otel/config.yaml create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp03 create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp04 create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp03 create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp04 create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/install-sb.ps1 create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/sb-central.env create mode 100644 docs/deploy/2026-11-19-prod-stage/sb/sb-site.env create mode 100644 docs/deploy/2026-11-19-prod-stage/sql/sb-role-mappings-prod.sql create mode 100644 docs/deploy/2026-11-19-prod-stage/sql/scadabridge-schema-2e0ce81e.sql create mode 100644 docs/deploy/2026-11-19-prod-stage/sql/zbsecrets-schema-v1.sql diff --git a/docs/deploy/2026-11-19-prod-stage/README.md b/docs/deploy/2026-11-19-prod-stage/README.md new file mode 100644 index 0000000..d713136 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/README.md @@ -0,0 +1,95 @@ +# 2026-11-19: PROD install pack staged on wonder-app-vp03/vp04 + wonder-par-vp03/vp04 (SMB only) + +**Owner request (2026-11-19):** "prep the application files, scripts, configs and copy them to the 4 servers for prod". +Plan: [`docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md`](../../plans/2026-11-19-prod-scadabridge-mxgw-install.md). + +**What was done:** files copied over SMB (445) as `NAM\adm_dohertj2`. **Nothing ran on any PROD box**: no WinRM, no +service, no install, no KEK, no secret. The PROD deploy hold and every phase gate in the plan still apply. Each script +below is run on the box only after the owner's go for its phase. + +## Artifacts (the QA-soaked builds, byte-identical; plan D-P2) + +| File | sha256 | Source | +|---|---|---| +| `scadabridge-2e0ce81e.zip` (app + cli, self-contained win-x64) | `6c706b0a…6572e23` | the zip QA vq03/vq04 run since 2026-11-19 | +| `mxgw-e74918cc.zip` (Server + x86 Worker `MxGateway.Worker.exe`, `0.1.2+e74918cc`) | `f0999b75…5d847f2` | the zip DEV vd03 + QA run | +| `otelcol-contrib_0.162.0_windows_amd64.zip` | `97a675ee…e496a167` | QA vq03 `E:\psrem-stage\otel-54` (scadaproj#54) | +| Secrets CLI 0.6.6 `ZB.MOM.WW.Secrets.Cli.exe` | `36dbd512…` | = QA `E:\ApiInstall\_tools\secret-0.6.6` | +| Secrets CLI 0.6.6 + SqlServer provider (centrals only) | `421f95e9…` | = QA `…\secret-0.6.6-sql` (QA finding 2) | +| `nssm.exe` | `eee9c44c…` | = QA `C:\ProgramData\ZB\qa-build\tools\nssm.exe` | + +Every staged folder carries a `SHA256SUMS.txt`. Remote sizes were checked file by file over SMB (150/150), and the two +app zips plus the CLI were pulled back from vp04 and par-vp04 and re-hashed. + +## Layout on each box + +| Path | vp03 / vp04 | par-vp03 / par-vp04 | +|---|---|---| +| `E:\psrem-stage\prod-common\` | `nssm.exe`, `preflight-prod.ps1`, `seed-secret-prod.ps1` | same | +| `E:\psrem-stage\mxgw-e74918cc\` | zip, `appsettings.json.prod`, `mxgw.env`, `install-mxgw.ps1`, `Install-`/`Upgrade-MxGatewayService.ps1` | same (install refuses until MXAccess exists, Q3) | +| `E:\psrem-stage\sb-2e0ce81e\` | zip, `appsettings.Central.json.prod-`, `sb-central.env`, `install-sb.ps1`, `sql\` | zip, `appsettings.Site.json.prod-`, `sb-site.env`, `install-sb.ps1` | +| `E:\psrem-stage\otel-54\` | collector zip, `config.yaml`, `Install-OtelCollector.ps1` (PROD tier added), `Test-OtelCollector.ps1` | same | +| `E:\ApiInstall\_tools\` | `secret-0.6.6\`, `secret-0.6.6-sql\` | `secret-0.6.6\` | + +`E:\ApiInstall` now exists only for `_tools`; there is still no `app` folder and no service, so both install scripts see a +greenfield box. + +## Configs (sources in this folder) + +- **mxgw** `mxgw.env`: QA vq03's key set plus the §3A.3 LDAP block. Differences: `ASPNETCORE_ENVIRONMENT=Production` (login + hard-stops on), `_PROD` `GroupToRole`, and the file KEK `C:\ProgramData\ZB\kek\mxgateway.kek`. `appsettings.json.prod` = the + zip's `Server\appsettings.json` minus `Dashboard:GroupToRole` (`GwAdmin`/`GwReader`). +- **ScadaBridge centrals**: per node `appsettings.Central.json.prod-vp03|vp04` (QA shape, IPv4 literals, self-first seeds, + `leader-auto-down`, BootstrapGuard on, NAM LDAPS block, `RequireHttpsCookie=false`, shared SQL secret + API-key stores, + CentralGrpcPort left at the default **8083**: QA's 8086 only existed because of co-location). `sb-central.env` is QA's key + set minus `MachineDataDb` (gone since 3808b49f), minus the LDAP password env (§3A.3), and minus `SitePsks` + (central reads `SB-GRPC-PSK-` from its store, Phase 6). +- **ScadaBridge sites**: `appsettings.Site.json.prod-parvp03|parvp04` (node-a/node-b, `PeerAddress` on node-a only, both + centrals at `:8083`, absolute data paths). `sb-site.env` = QA's site key set. +- **SQL (centrals, `sql\`)**: `scadabridge-schema-2e0ce81e.sql` (full, non-idempotent, 65 migrations, from `dotnet ef + migrations script 0 20261115221558_RemoveAuditLogHashChain` at 2e0ce81e; run with `sqlcmd -I -b`), + `zbsecrets-schema-v1.sql`, `sb-role-mappings-prod.sql` (the five `_PROD` rows, D-P6). + +### Placeholders still open (the install scripts refuse while any remains) + +| Placeholder / value | Where | Question | +|---|---|---| +| `__Q2_ALARM_EXPRESSION__` | `mxgw.env` | Q2 alarm area (`\\\Galaxy!`) | +| `MxGateway__Galaxy__ConnectionString` (value file) | install time | Q2 prod GR SQL | +| `__Q8_SITE_ID__` | both `appsettings.Site.json.prod-*` | Q8 SiteId | +| `ScadaBridge__Database__ConfigurationDb`, `Secrets__SqlServer__ConnectionString` (value files) | install time | Q1 prod SQL + `scadabridge_svc` | +| JWT key, inbound pepper (same on both centrals), mxgw pepper (shared by the PAR pair, pattern a), site PSK + LocalDb key (same on both sites) | install time, value files | generated then, stored in the Keychain | + +When a placeholder is resolved, edit the file here, commit, and re-push that one file (its `SHA256SUMS.txt` line too). + +## Scripts (dry first, always) + +| Script | Does | Never does | +|---|---|---| +| `prod-common\preflight-prod.ps1 [-SqlHost] [-GrHost]` | read-only Phase 2 gates: disk, `LmxProxy.dll` + COM registration, NAM 636/389 + rootca + TLS, greenfield paths and ports, DNS, TCP path probes per §4 (open / refused / timeout), staged hashes | write anything | +| `mxgw-e74918cc\install-mxgw.ps1 -Mode dry\|real [-ValuesDir] [-KekFrom]` | MXAccess gate, zip hash, placeholder + `_DEV/_QA` refusal, expand to `app`/`worker`, NAM base config, 32-byte KEK, NSSM service via the patched `Install-MxGatewayService.ps1` | start the service | +| `sb-2e0ce81e\install-sb.ps1 -Role Central\|Site -Mode dry\|real [-ValuesDir] [-KekFrom]` | box/role check, zip hash, this node's config, env from value files, expand `app`/`cli`, `\{data,logs}`, KEK (generated on vp03 only; every other node takes the same file), `sc.exe` service with recovery 5/5/10 s + failureflag | start the service | +| `prod-common\seed-secret-prod.ps1 -Target MxAccessGw\|ScadaBridge-Central -ValueFile` | seeds `nam-ldap-svc-password` into a stopped service's store (central: once, vp03, shared SQL store), reads back, deletes the value file | print the value | + +`Install-MxGatewayService.ps1` here is mxgw `e74918cc` plus three marked patches: `-WorkerExeName` (the wonder Worker is +`MxGateway.Worker.exe`), File-KEK acceptance, and masking of connection strings in its env printout (the stock script +prints `MxGateway__Galaxy__ConnectionString` in clear). `Upgrade-MxGatewayService.ps1` is the `START_PENDING`-patched copy QA +used. Both repo fixes are still owed (mxaccessgw). + +Checked on the Mac (pwsh 7, stubs for `nssm`/`icacls`): all eight scripts parse; `install-sb.ps1` dry runs pass for a +central and a site and refuse a wrong role and an open `__Q8_SITE_ID__`; `install-mxgw.ps1` dry run refuses an open `__Q2__` +and then runs `Install-MxGatewayService -DryRun` (39 env entries). That is not a Windows run; the on-box dry run is the gate. + +## Order once the owner says go (plan phases) + +1. Phase 2, each box: `preflight-prod.ps1` (needs the Q6 go for remote exec). Optional: `otel-54\Install-OtelCollector.ps1 -StageDir … -Tier PROD -NssmPath E:\psrem-stage\prod-common\nssm.exe`. +2. Phase 3 (Q1 answered): `sql\scadabridge-schema-2e0ce81e.sql`, `sql\zbsecrets-schema-v1.sql`, `sql\sb-role-mappings-prod.sql`; backup. +3. Phase 4, vp03/vp04 (par later, Q3): `install-mxgw.ps1 -Mode dry`, then `real`; `seed-secret-prod.ps1 -Target MxAccessGw`; start; §3A.5. +4. Phase 5: `install-sb.ps1 -Role Central` on vp03 (KEK generated), vp04 (`-KekFrom` the vp03 file); `seed-secret-prod.ps1 -Target ScadaBridge-Central` on vp03; start vp03, then vp04; §3A.5. +5. Phase 6: `install-sb.ps1 -Role Site` on both PAR boxes (`-KekFrom`, same PSK and LocalDb key); `SB-GRPC-PSK-` into central; start both together. + +## Cleanup (if PROD is abandoned) + +`winrm smb rm -r 'E$\psrem-stage\prod-common' 'E$\psrem-stage\mxgw-e74918cc' 'E$\psrem-stage\sb-2e0ce81e' 'E$\psrem-stage\otel-54' 'E$\ApiInstall\_tools'` +on each box. The push also created the parent folders `E:\psrem-stage` and `E:\ApiInstall`; the earlier 2026-11-19 +SMB survey listed neither. Nothing else was created. diff --git a/docs/deploy/2026-11-19-prod-stage/common/preflight-prod.ps1 b/docs/deploy/2026-11-19-prod-stage/common/preflight-prod.ps1 new file mode 100644 index 0000000..ffa6696 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/common/preflight-prod.ps1 @@ -0,0 +1,65 @@ +# PROD box preflight (plan Phase 2 gates + S3A LDAP gate). READ-ONLY: writes nothing, starts nothing, prints no secret. +# Run on each of the four boxes after the owner's go for remote exec (Q6). Optional -SqlHost/-GrHost for Q1/Q2 reachability. +param([string]$SqlHost, [string]$GrHost) +$ErrorActionPreference = 'Continue'; $ProgressPreference = 'SilentlyContinue' +$box = $env:COMPUTERNAME.ToUpperInvariant() +"== preflight-prod $box $(Get-Date -Format s)" +$ip = @{ 'WONDER-APP-VP03' = '10.220.153.205'; 'WONDER-APP-VP04' = '10.220.153.210'; 'WONDER-PAR-VP03' = '10.201.75.86'; 'WONDER-PAR-VP04' = '10.201.75.87' } +$central = $box -like 'WONDER-APP-*' +"role (planned): $(if ($central) { 'ScadaBridge Central + mxgw' } else { 'ScadaBridge Site + mxgw' })" +Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" | ForEach-Object { "disk $($_.DeviceID) free $([math]::Round($_.FreeSpace/1GB,1)) GB of $([math]::Round($_.Size/1GB,1)) GB" } +"os $((Get-CimInstance Win32_OperatingSystem).Caption) ps $($PSVersionTable.PSVersion)" + +# MXAccess (Q3 gate for mxgw) +$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll' +if (Test-Path $lmx) { "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12)) (want 6000.0045.2657.1 / BD75A2FDE222)" } else { 'LmxProxy.dll ABSENT - mxgw cannot run here yet (Q3)' } +$clsid = Get-ItemProperty 'Registry::HKEY_CLASSES_ROOT\WOW6432Node\CLSID\*\ProgID' -ErrorAction SilentlyContinue | Where-Object { $_.'(default)' -like 'LMXProxy.LMXProxyServer*' } +"LMXProxy.LMXProxyServer COM registration: $(if ($clsid) { 'present' } else { 'absent' })" + +# NAM LDAPS (S3A gate) +foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') { foreach ($p in 636,389) { + $c = New-Object Net.Sockets.TcpClient; $ok = $c.ConnectAsync($h,$p).Wait(4000); $c.Dispose(); "reach $h`:$p $ok" } } +$root = @(Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like '*zimmer-rootca*' }) +"zimmer-rootca in LocalMachine\Root: $($root.Count) $(($root | ForEach-Object { $_.Thumbprint + ' exp ' + $_.NotAfter.ToString('yyyy-MM-dd') }) -join '; ')" +foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') { + try { $t = New-Object Net.Sockets.TcpClient($h,636); $script:err = $null + $s = New-Object Net.Security.SslStream($t.GetStream(), $false, { param($a,$b,$c,$e) $script:err = $e; $true }) + $s.AuthenticateAsClient($h); "tls $h policyErrors=$script:err proto=$($s.SslProtocol)"; $s.Dispose(); $t.Dispose() } catch { "tls $h FAILED $($_.Exception.Message)" } } + +# Local state: greenfield expected +foreach ($d in 'E:\ApiInstall','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\ScadaBridge','C:\ProgramData\ZB\kek') { "path $d exists=$(Test-Path $d)" } +Get-Service | Where-Object { $_.Name -match 'MxAccess|MxGateway|ScadaBridge|ZbOtel|OtOpcUa|Historian' } | ForEach-Object { "service $($_.Name) $($_.Status)" } +$listen = Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue | Where-Object { $_.LocalPort -in 4317,5120,5130,8081,8082,8083,8084,8085,8889 } +foreach ($l in $listen) { "PORT IN USE $($l.LocalAddress):$($l.LocalPort) pid $($l.OwningProcess) $((Get-Process -Id $l.OwningProcess -ErrorAction SilentlyContinue).ProcessName)" } +if (-not $listen) { 'ports 4317 5120 5130 8081-8085 8889: all free' } +$fw = Get-NetFirewallProfile -ErrorAction SilentlyContinue | ForEach-Object { "$($_.Name)=$($_.Enabled)" } +"windows firewall profiles: $($fw -join ' ')" + +# Name resolution + paths to the other boxes (plan S4). open = listener; refused = path open, nothing listening yet; +# timeout = probably filtered. Nothing listens yet on a greenfield box, so 'refused' is the good answer here. +function Probe($h, $p) { + $c = New-Object Net.Sockets.TcpClient + try { $ok = $c.ConnectAsync($h, $p).Wait(3000); if ($ok) { 'open' } else { 'timeout' } } + catch { $m = $_.Exception.InnerException; while ($m.InnerException) { $m = $m.InnerException }; if ($m.Message -match 'refused|actively') { 'refused' } else { "error: $($m.Message)" } } + finally { $c.Dispose() } } +foreach ($n in 'wonder-app-vp03','wonder-app-vp04','wonder-par-vp03','wonder-par-vp04') { + $f = "$n.zmr.zimmer.com"; try { $a = ([Net.Dns]::GetHostAddresses($f) | Select-Object -First 1).IPAddressToString } catch { $a = 'NXDOMAIN' } + "dns $f -> $a (plan $($ip[$n.ToUpperInvariant()]))" } +$targets = if ($central) { + @(@{h = $ip[($(if ($box -eq 'WONDER-APP-VP03') {'WONDER-APP-VP04'} else {'WONDER-APP-VP03'}))]; p = 8081, 8085, 8083}, + @{h = '10.201.75.86'; p = 8083, 5120}, @{h = '10.201.75.87'; p = 8083, 5120}) +} else { + @(@{h = $ip[($(if ($box -eq 'WONDER-PAR-VP03') {'WONDER-PAR-VP04'} else {'WONDER-PAR-VP03'}))]; p = 8082, 8083, 5120}, + @{h = '10.220.153.205'; p = 8083, 5120}, @{h = '10.220.153.210'; p = 8083, 5120}) +} +foreach ($t in $targets) { foreach ($p in $t.p) { "path -> $($t.h):$p $(Probe $t.h $p)" } } +if ($SqlHost) { "path -> SQL $SqlHost`:1433 $(Probe $SqlHost 1433)" } +if ($GrHost) { "path -> GR $GrHost`:1433 $(Probe $GrHost 1433)" } + +# Staged artifacts +foreach ($sums in Get-ChildItem 'E:\psrem-stage' -Recurse -Filter 'SHA256SUMS.txt' -ErrorAction SilentlyContinue) { + foreach ($line in Get-Content $sums.FullName) { if ($line -match '^([0-9a-fA-F]{64})\s+\*?(.+)$') { + $f = Join-Path $sums.DirectoryName $Matches[2] + if (-not (Test-Path $f)) { "stage MISSING $f" } elseif ((Get-FileHash $f).Hash -ne $Matches[1].ToUpperInvariant()) { "stage HASH MISMATCH $f" } else { "stage ok $f" } } } } +foreach ($e in 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe','E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe') { if (Test-Path $e) { "tool $e sha256 $((Get-FileHash $e).Hash.Substring(0,12))" } } +"PREFLIGHT DONE" diff --git a/docs/deploy/2026-11-19-prod-stage/common/seed-secret-prod.ps1 b/docs/deploy/2026-11-19-prod-stage/common/seed-secret-prod.ps1 new file mode 100644 index 0000000..fb2c5e9 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/common/seed-secret-prod.ps1 @@ -0,0 +1,56 @@ +# Seed nam-ldap-svc-password into this PROD box's stores BEFORE each service's first start (plan S3A.4). QA seed-secret.ps1 +# retargeted. Each store's location and KEK come from the installed (stopped) service's own env, never assumed. +# -Target MxAccessGw : this box's mxgw SQLite store (all four boxes) +# -Target ScadaBridge-Central : the SHARED SQL Server store (zbsecrets), ONCE, on vp03 only, with the SQL-aware CLI +# (the headless 0.6.6 CLI writes SQLite only). Needs sql\zbsecrets-schema-v1.sql applied. +# The value arrives as -ValueFile over SMB (Keychain -> 0600 temp -> SMB), is ACL'd, read back, compared, then DELETED. +# It is never printed and never on argv. +param( + [Parameter(Mandatory)][ValidateSet('MxAccessGw','ScadaBridge-Central')][string]$Target, + [Parameter(Mandatory)][string]$ValueFile +) +$ErrorActionPreference = 'Stop' +$cli = 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe' +$cliSql = 'E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe' +$name = 'nam-ldap-svc-password' +$desc = 'NAM LDAP lookup account (svc_igntadmin, PROD interim D-P5)' +& icacls.exe $ValueFile /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null +function SvcEnv($svc) { + $k = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc" + if (-not (Test-Path $k)) { throw "service $svc is not installed" } + $l = @((Get-ItemProperty $k -ErrorAction SilentlyContinue).Environment) + @((Get-ItemProperty "$k\Parameters" -ErrorAction SilentlyContinue).AppEnvironmentExtra) | Where-Object { $_ } + $h = @{}; foreach ($e in $l) { $i = $e.IndexOf('='); if ($i -gt 0) { $h[$e.Substring(0,$i)] = $e.Substring($i+1) } }; $h +} +$names = 'Secrets__SqlitePath','Secrets__MasterKey__Source','Secrets__MasterKey__FilePath','Secrets__MasterKey__EnvVarName','Secrets__Store__Provider','Secrets__SqlServer__ConnectionString','ZB_SECRETS_MASTER_KEY' +$fail = 0 +try { + if ($Target -eq 'ScadaBridge-Central' -and $env:COMPUTERNAME -ne 'WONDER-APP-VP03') { throw 'the shared central store is seeded once, from vp03' } + $svc = Get-Service $Target; if ($svc.Status -ne 'Stopped') { throw "$Target is $($svc.Status); seed before the first start" } + $want = [IO.File]::ReadAllText($ValueFile).TrimEnd("`r", "`n") + if (-not $want) { throw 'value file is empty' } + "value file read, length $($want.Length)" + $e = SvcEnv $Target + foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue; if ($e.ContainsKey($n)) { Set-Item "Env:$n" $e[$n] } } + $exe = $cli + if ($Target -eq 'ScadaBridge-Central') { + $env:Secrets__Store__Provider = 'SqlServer'; $exe = $cliSql + if (-not $env:Secrets__SqlServer__ConnectionString) { throw 'Central env has no Secrets__SqlServer__ConnectionString' } + } + if ($env:Secrets__MasterKey__Source -ne 'File') { throw "master key source '$($env:Secrets__MasterKey__Source)' is not File" } + if (-not (Test-Path $env:Secrets__MasterKey__FilePath)) { throw "KEK $($env:Secrets__MasterKey__FilePath) missing" } + $where = if ($Target -eq 'ScadaBridge-Central') { 'SqlServer shared store' } else { $env:Secrets__SqlitePath } + $tmp = "$ValueFile.nl"; [IO.File]::WriteAllText($tmp, $want) + & icacls.exe $tmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null + $out = & cmd.exe /c "`"$exe`" set $name - --description `"$desc`" < `"$tmp`"" 2>&1 + $rc = $LASTEXITCODE + Remove-Item $tmp -Force -ErrorAction SilentlyContinue + $got = (& $exe get $name 2>$null | Out-String).TrimEnd("`r", "`n") + $match = ($got -ceq $want) + if (-not $match) { try { $j = $got | ConvertFrom-Json; $match = ($j.value -ceq $want) } catch {} } + "$Target -> $where kek=$($env:Secrets__MasterKey__FilePath) kek-sha256=$((Get-FileHash $env:Secrets__MasterKey__FilePath).Hash.Substring(0,8)) set rc=$rc out=$(($out | Out-String).Trim()) readback-match=$match" + if ($rc -ne 0 -or -not $match) { $fail++ } +} finally { + Remove-Item $ValueFile, "$ValueFile.nl" -Force -ErrorAction SilentlyContinue; "value file removed: $(-not (Test-Path $ValueFile))" + foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue } +} +"SEED DONE failures=$fail"; exit $fail diff --git a/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1 b/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1 new file mode 100644 index 0000000..8e55a6d --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/mxgw/Install-MxGatewayService.ps1 @@ -0,0 +1,213 @@ +#requires -Version 5.1 +# PROD STAGE COPY (2026-11-19): mxaccessgw e74918cc scripts/deploy/Install-MxGatewayService.ps1 + 3 patches marked PROD-STAGE PATCH. +<# +.SYNOPSIS + On-box NEW-BOX install of the MxAccess Gateway as an NSSM service (mxaccessgw#220). + Companion of Upgrade-MxGatewayService.ps1; see docs/runbooks/DeployUpgradeNssmService.md. + +.DESCRIPTION + RUN ON THE WINDOWS BOX (Windows PowerShell 5.1). Creates the service ONCE, with: + * Application / AppDirectory / DisplayName / Description, auto start, account (default LocalSystem); + * NSSM exit handling: restart on any exit with a throttle, graceful-stop timeouts that cover the + Worker shutdown grace (MxGateway:Worker:ShutdownTimeoutSeconds, default 10 s); + * stdout/stderr redirection with rotation; + * Windows SCM recovery (sc failure: restart, restart, restart + reset period); + * the environment block, set in ONE `nssm set AppEnvironmentExtra` call on a service that has none + (so there is nothing to wipe). The env comes from -EnvFile (KEY=VALUE lines, # comments) and/or + -Env (hashtable); -Env wins. Secret VALUES are never printed (dry-run shows names and masks values). + It does NOT start the service unless -Start is given (review the env first), and it refuses to run if + the service already exists (use Upgrade-MxGatewayService.ps1) or if any gateway process is running + (OrphanWorkerTerminator trap: a second gateway kills live Workers). + + Required env (checked by name): ZB_SECRETS_MASTER_KEY, a *Pepper* entry (MxGateway__ApiKeyPepper), + MxGateway__Worker__ExecutablePath (defaults to \ZB.MOM.WW.MxGateway.Worker.exe), + MxGateway__Authentication__SqlitePath and Secrets__SqlitePath (default under -DataDir; must be absolute and + OUTSIDE the content root, the validator is fail-closed), ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT. + Kestrel endpoints default to docs/GatewayConfiguration.md's cleartext h2c :5120 + dashboard :5130 unless + the env supplies Kestrel__Endpoints__*. + + -DryRun (or -WhatIf) prints every mutating action and performs none. +#> +[CmdletBinding(SupportsShouldProcess = $true)] +param( + [Parameter(Mandatory = $true)][string]$ServerDir, + [Parameter(Mandatory = $true)][string]$WorkerDir, + [string]$ServiceName = 'MxAccessGw', + [string]$NssmPath = 'nssm', + [string]$DataDir = 'C:\ProgramData\MxGateway', + [string]$LogDir, + [string]$EnvFile, + [hashtable]$Env = @{}, + [string]$ServerExeName = 'ZB.MOM.WW.MxGateway.Server.exe', + # PROD-STAGE PATCH 1: the wonder Worker build is named MxGateway.Worker.exe, not ZB.MOM.WW.MxGateway.Worker.exe. + [string]$WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe', + [string]$ObjectName = 'LocalSystem', + [string]$ServiceAccountPassword, + [string]$EnvironmentName = 'Production', + [int]$RestartDelayMs = 5000, + [int]$ThrottleMs = 10000, + [int]$StopGraceMs = 20000, + [int]$RotateBytes = 10485760, + [switch]$Start, + [switch]$DryRun +) + +$ErrorActionPreference = 'Stop' +$script:Dry = $DryRun.IsPresent -or $WhatIfPreference +$ServerProc = 'ZB.MOM.WW.MxGateway.Server' +$WorkerProc = 'ZB.MOM.WW.MxGateway.Worker' + +function Say([string]$m) { Write-Host $m } +function Fail([string]$m) { throw "ABORT: $m" } +function Step([string]$Description, [scriptblock]$Action) { + if ($script:Dry) { Say "[DRYRUN] $Description"; return } + Say "[ACTION] $Description"; & $Action +} +function Invoke-Nssm { + param([string[]]$NssmArgs, [switch]$AllowFail) + $raw = & $NssmPath @NssmArgs 2>&1; $code = $LASTEXITCODE + $text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", '' + if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs[0..1] -join ' ') failed (exit $code): $text" } + [pscustomobject]@{ Code = $code; Text = $text.Trim() } +} +function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') } +function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') } +function Test-Inside([string]$Path, [string]$Root) { + $a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant() + $a -eq $b -or $a.StartsWith("$b/") +} +function Mask([string]$k, [string]$v) { + # PROD-STAGE PATCH 3: connection strings can carry a SQL password; mask them too. + if ($k -match 'KEY|PASSWORD|SECRET|PEPPER|TOKEN|CONNECTIONSTRING' -and $k -notmatch 'Path$|Url$') { '****' } else { $v } +} + +try { + Say "Install-MxGatewayService service=$ServiceName dry=$($script:Dry)" + + # --- environment assembly ------------------------------------------------------------------------- + $envMap = [ordered]@{} + if ($EnvFile) { + if (-not (Test-Path -LiteralPath $EnvFile)) { Fail "-EnvFile '$EnvFile' not found" } + foreach ($line in (Get-Content -LiteralPath $EnvFile)) { + $t = $line.Trim() + if (-not $t -or $t.StartsWith('#')) { continue } + if ($t -match '^([^=\s]+)=(.*)$') { $envMap[$Matches[1]] = $Matches[2] } else { Fail "-EnvFile line is not KEY=VALUE: '$t'" } + } + } + foreach ($k in $Env.Keys) { $envMap[[string]$k] = [string]$Env[$k] } + + $workerExe = Join-Path $WorkerDir $WorkerExeName + if (-not $envMap.Contains('MxGateway__Worker__ExecutablePath')) { $envMap['MxGateway__Worker__ExecutablePath'] = $workerExe } + if (-not $envMap.Contains('MxGateway__Authentication__SqlitePath')) { $envMap['MxGateway__Authentication__SqlitePath'] = Join-Path $DataDir 'gateway-auth.db' } + if (-not $envMap.Contains('Secrets__SqlitePath')) { $envMap['Secrets__SqlitePath'] = Join-Path $DataDir 'mxgateway-secrets.db' } + if (-not $envMap.Contains('ASPNETCORE_ENVIRONMENT')) { $envMap['ASPNETCORE_ENVIRONMENT'] = $EnvironmentName } + if (-not $envMap.Contains('DOTNET_ENVIRONMENT')) { $envMap['DOTNET_ENVIRONMENT'] = $EnvironmentName } + if (-not ($envMap.Keys | Where-Object { $_ -like 'Kestrel__Endpoints__*' })) { + $envMap['Kestrel__Endpoints__Http__Url'] = 'http://0.0.0.0:5120' + $envMap['Kestrel__Endpoints__Http__Protocols'] = 'Http2' + $envMap['Kestrel__Endpoints__Dashboard__Url'] = 'http://0.0.0.0:5130' + Say '[info] no Kestrel__Endpoints__* supplied: using the documented cleartext defaults (5120 h2c, 5130 dashboard)' + } + + # --- validation ------------------------------------------------------------------------------------- + # PROD-STAGE PATCH 2: accept a file KEK (Secrets__MasterKey__Source=File + FilePath), the QA/PROD pattern. + if ($envMap['Secrets__MasterKey__Source'] -eq 'File') { + $kf = $envMap['Secrets__MasterKey__FilePath'] + if (-not $kf -or -not (Test-Rooted $kf)) { Fail 'Secrets__MasterKey__Source=File needs an absolute Secrets__MasterKey__FilePath' } + if (-not $script:Dry -and -not (Test-Path -LiteralPath $kf)) { Fail "KEK file '$kf' not found" } + } elseif (-not $envMap.Contains('ZB_SECRETS_MASTER_KEY') -or -not $envMap['ZB_SECRETS_MASTER_KEY']) { Fail 'env must supply ZB_SECRETS_MASTER_KEY (the Secrets KEK) or Secrets__MasterKey__Source=File' } + if (-not ($envMap.Keys | Where-Object { $_ -match 'Pepper' })) { Fail 'env must supply the API-key pepper (MxGateway__ApiKeyPepper); without it no issued token verifies' } + foreach ($k in 'MxGateway__Authentication__SqlitePath', 'Secrets__SqlitePath') { + $v = $envMap[$k] + if (-not (Test-Rooted $v)) { Fail "$k '$v' must be absolute" } + if ((Test-Inside $v $ServerDir) -or (Test-Inside $v $WorkerDir)) { Fail "$k '$v' is inside the content root; the validator is fail-closed and an upgrade swap would lose it" } + } + foreach ($d in @($ServerDir, $WorkerDir)) { if (-not (Test-Rooted $d)) { Fail "'$d' must be an absolute path" } } + if ((Test-Inside $WorkerDir $ServerDir) -or (Test-Inside $ServerDir $WorkerDir)) { Fail 'Server and Worker directories must be siblings, not nested (the upgrade swap renames them independently)' } + if (-not (Test-Path -LiteralPath (Join-Path $ServerDir $ServerExeName))) { Fail "$ServerExeName not found in $ServerDir" } + if (-not (Test-Path -LiteralPath $workerExe)) { Fail "$WorkerExeName not found in $WorkerDir" } + if ($envMap['MxGateway__Worker__ExecutablePath'] -ne $workerExe) { Say '[WARN] MxGateway__Worker__ExecutablePath differs from \Worker.exe; keep ONE canonical spelling' } + if ($envMap['ASPNETCORE_ENVIRONMENT'] -ne 'Production') { Say '[WARN] ASPNETCORE_ENVIRONMENT is not exactly Production: DisableLogin / plaintext-LDAP hard-stops are only enforced for Production' } + + # --- one-gateway guard + existence ------------------------------------------------------------------- + $st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail + if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" } + # Both naming schemes (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes) plus the exe being installed (#230). + $gwImages = @($ServerProc, $WorkerProc, 'MxGateway.Server', 'MxGateway.Worker', [IO.Path]::GetFileNameWithoutExtension($ServerExeName)) | Select-Object -Unique + $procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in $gwImages }) + if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" } + + if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' } + $exe = Join-Path $ServerDir $ServerExeName + + Say "[info] env entries ($($envMap.Count)):" + foreach ($k in $envMap.Keys) { Say " $k=$(Mask $k $envMap[$k])" } + + # --- install ------------------------------------------------------------------------------------------------ + Step "create $DataDir and $LogDir (stores/certs/logs live OUTSIDE the content root)" { + New-Item -ItemType Directory -Force -Path $DataDir, $LogDir | Out-Null + } + Step "nssm install $ServiceName `"$exe`"" { [void](Invoke-Nssm -NssmArgs @('install', $ServiceName, $exe)); $script:Installed = $true } + $settings = @( + @('AppDirectory', $ServerDir), + @('DisplayName', 'MxAccess Gateway'), + @('Description', 'MxAccess Gateway (gRPC) - Server; launches the x86 Worker per session'), + @('Start', 'SERVICE_AUTO_START'), + @('Type', 'SERVICE_WIN32_OWN_PROCESS'), + @('AppExit', 'Default', 'Restart'), + @('AppRestartDelay', "$RestartDelayMs"), + @('AppThrottle', "$ThrottleMs"), + @('AppStopMethodConsole', "$StopGraceMs"), + @('AppStopMethodWindow', '0'), + @('AppStopMethodThreads', '0'), + @('AppStdout', (Join-Path $LogDir 'service-stdout.log')), + @('AppStderr', (Join-Path $LogDir 'service-stderr.log')), + @('AppRotateFiles', '1'), + @('AppRotateOnline', '1'), + @('AppRotateBytes', "$RotateBytes") + ) + foreach ($s in $settings) { + Step "nssm set $ServiceName $($s -join ' ')" { [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName) + $s)) } + } + if ($ObjectName -and $ObjectName -ne 'LocalSystem') { + Step "nssm set $ServiceName ObjectName $ObjectName " { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', $ObjectName, $ServiceAccountPassword)) } + } else { + Step "nssm set $ServiceName ObjectName LocalSystem" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', 'LocalSystem')) } + } + + # Env block: first and only AppEnvironmentExtra write; the service has none, so nothing is wiped. + $pairs = @($envMap.Keys | ForEach-Object { "$_=$($envMap[$_])" }) + Step "nssm set $ServiceName AppEnvironmentExtra <$($envMap.Count) entries, values hidden>" { + [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName, 'AppEnvironmentExtra') + $pairs)) + } + + # SCM recovery: restart x3 (5 s, 5 s, 30 s), reset the counter after 1 day; also on non-crash exits. + Step "sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/30000" { + & sc.exe failure $ServiceName reset= 86400 actions= 'restart/5000/restart/5000/restart/30000' | Out-Null + if ($LASTEXITCODE -ne 0) { Fail "sc.exe failure exited $LASTEXITCODE" } + } + Step "sc.exe failureflag $ServiceName 1" { + & sc.exe failureflag $ServiceName 1 | Out-Null + if ($LASTEXITCODE -ne 0) { Fail "sc.exe failureflag exited $LASTEXITCODE" } + } + + # Verify read-back (names only) so a silent miss is caught before the first start. + Step "verify: nssm get AppEnvironmentExtra has $($envMap.Count) entries" { + $txt = (Invoke-Nssm -NssmArgs @('get', $ServiceName, 'AppEnvironmentExtra')).Text + $n = @($txt -split "[`r`n]+" | Where-Object { $_.Trim() -match '^[^=\s]+=' }).Count + if ($n -ne $envMap.Count) { Fail "env read-back has $n entries, expected $($envMap.Count)" } + } + + if ($Start) { + Step "nssm start $ServiceName (exactly once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) } + Say 'Next: GET /health/ready on the dashboard port, then prove a Worker session with an API key (runbook section 7).' + } else { + Say '[info] service installed STOPPED. Review the env, then `nssm start` (or re-run with -Start). Seed secrets with the `secret` CLI using the SAME Secrets__SqlitePath + ZB_SECRETS_MASTER_KEY.' + } + Say '== DONE ==' + exit 0 +} catch { + Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red + if ($script:Installed) { Write-Host "PARTIAL INSTALL: inspect, then `nssm remove $ServiceName confirm` and re-run." -ForegroundColor Yellow } + exit 1 +} diff --git a/docs/deploy/2026-11-19-prod-stage/mxgw/Upgrade-MxGatewayService.ps1 b/docs/deploy/2026-11-19-prod-stage/mxgw/Upgrade-MxGatewayService.ps1 new file mode 100644 index 0000000..f4538f4 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/mxgw/Upgrade-MxGatewayService.ps1 @@ -0,0 +1,869 @@ +#requires -Version 5.1 +<# +.SYNOPSIS + On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker). + Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230). + +.DESCRIPTION + RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the + new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA). + + Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start -> + verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if + start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store + restore -> start -> verify. + + Safety properties: + * Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes + (NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes + (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by + exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an + abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes). + * Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is + rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are + never printed) at preflight, after the stop, after the swap, after the start and after the stability + window. Any added, removed or changed entry aborts (and triggers the automatic rollback). + * Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json, + dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified. + * API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are + only COPIED (db + -wal + -shm) into *.bak- siblings while the service is stopped. + * A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade). + + -DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state, + registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real. + +.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application). +.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath). +.PARAMETER ServiceName NSSM service name. Default MxAccessGw. +.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH). +.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory. +.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath. +.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify) + must carry it in ProductVersion ('0.1.2+'). Builds from 2026-07-09..08-10 have a bogus stamp. +.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites'). +.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous). +.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp. +.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak. sets). +.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak- (schema-bump case). +.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails. +.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir. +.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one. +.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables. +.PARAMETER DryRun Print, do not act. -WhatIf is equivalent. +#> +[CmdletBinding(SupportsShouldProcess = $true)] +param( + [string]$ServerSource, + [string]$WorkerSource, + [string]$ServiceName = 'MxAccessGw', + [string]$NssmPath = 'nssm', + [string]$ServerDir, + [string]$WorkerDir, + [string]$ExpectedSha, + [string]$Label, + [string]$HealthBaseUrl = 'http://localhost:5130', + [switch]$Rollback, + [string]$BackupStamp, + [switch]$RestoreStores, + [switch]$NoAutoRollback, + [switch]$TakeNewAppSettings, + [switch]$AllowRuntimeDowngrade, + [switch]$SkipFreeSpaceCheck, + [double]$MinFreeGB = 2, + [int]$StopTimeoutSeconds = 30, + [int]$StableSeconds = 60, + [int]$ReadyTimeoutSeconds = 120, + [switch]$DryRun +) + +$ErrorActionPreference = 'Stop' +$script:Dry = $DryRun.IsPresent -or $WhatIfPreference +# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY +# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230. +$WhatIfPreference = $false +$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss' +# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe. +# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context. +$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server') +$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker') +$script:ServerImages = $script:KnownServerImages +$script:WorkerImages = $script:KnownWorkerImages +$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath' +# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one. +$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer') +# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build). +$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd') +$script:Notes = New-Object System.Collections.Generic.List[string] +$script:Redact = New-Object System.Collections.Generic.List[string] +$script:LockStream = $null +$script:BackupDir = $null +$script:EnvWarnOnly = $false + +# --- output (secret values are redacted from every line) ---------------------------------------------------- + +function Hide([string]$m) { + if ($null -eq $m) { return '' } + foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '') } } + $m +} +function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) } +function Info([string]$m) { Say "[info] $m" } +function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") } +function Fail([string]$m) { throw "ABORT: $(Hide $m)" } + +# Mutating step: print in dry-run, run otherwise. +function Step([string]$Description, [scriptblock]$Action) { + if ($script:Dry) { Say "[DRYRUN] $Description"; return } + Say "[ACTION] $Description" + & $Action +} + +# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) --------------------------- + +function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' } + +function Invoke-Nssm { + param([string[]]$NssmArgs, [switch]$AllowFail) + $raw = & $NssmPath @NssmArgs 2>&1 + $code = $LASTEXITCODE + $text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", '' + if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" } + [pscustomobject]@{ Code = $code; Text = $text.Trim() } +} + +function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text } + +function Get-ServiceState { + $r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail + if ($r.Code -ne 0) { return 'MISSING' } + ($r.Text -split "`n")[0].Trim() +} + +# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get. +function Read-EnvBlockLines { + $key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters" + if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) { + $p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue + if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) } + return @() + } + @((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() }) +} + +# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable). +function Get-ProcessTable { + if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) { + return @(Get-CimInstance Win32_Process | ForEach-Object { + [pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath } + }) + } + @(Get-Process -ErrorAction SilentlyContinue | ForEach-Object { + $p = $null; try { $p = $_.Path } catch { } + [pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p } + }) +} + +# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read. +function Get-ServiceHostPid { + if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null } + $s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue + if ($s -and $s.ProcessId) { return [int]$s.ProcessId } + $null +} + +# Other services whose command line launches the gateway exe. $null when the check is unavailable. +function Get-OtherGatewayServices([string]$ExeName) { + if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null } + # The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable'). + , @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name }) +} + +function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue } + +function Get-HttpStatus([string]$Url) { + try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode } + catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 } +} + +function Get-ProductSha([string]$Exe) { + try { + $pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion + if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() } + } catch { } + $null +} + +# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read. +function Get-ExeArchitecture([string]$Exe) { + try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null } +} + +# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails. +function Protect-Directory([string]$Path) { + if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return } + $null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1 + if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" } +} + +function Export-ServiceKey([string]$Destination) { + if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return } + $null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1 + if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" } +} + +function Get-DirSddl([string]$Path) { + if (-not (Test-IsWindowsHost)) { return $null } + try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null } +} + +# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries. +function Copy-ExplicitAcl([string]$From, [string]$To) { + if (-not (Test-IsWindowsHost)) { return } + try { + $a = Get-Acl -LiteralPath $From + if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) { + Set-Acl -LiteralPath $To -AclObject $a + Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)" + } + } catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" } +} + +# --- pure helpers --------------------------------------------------------------------------------------------- + +function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') } +function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') } +function Test-Inside([string]$Path, [string]$Root) { + if (-not $Path -or -not $Root) { return $false } + $a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant() + $a -eq $b -or $a.StartsWith("$b/") +} +function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) } +function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] } +function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) } + +function Test-ShaMatch([string]$Actual, [string]$Expected) { + if (-not $Actual -or -not $Expected) { return $false } + $e = $Expected.ToLowerInvariant() + $Actual.StartsWith($e) -or $e.StartsWith($Actual) +} + +function Get-TextHashPrefix([string]$Text) { + $sha = [Security.Cryptography.SHA256]::Create() + try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) } + finally { $sha.Dispose() } +} +function Get-FileSha([string]$Path) { + $sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path) + try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() } +} + +function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') } + +# name -> value. Every sensitive value is registered for redaction. +function ConvertTo-EnvMap([string[]]$Lines) { + $map = [ordered]@{} + foreach ($line in $Lines) { + $i = $line.IndexOf('=') + if ($i -le 0) { continue } + $k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1) + $map[$k] = $v + if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) } + } + $map +} + +# Names + per-value SHA-256 prefixes. Never carries a value. +function Get-EnvFingerprint([string[]]$Lines) { + $map = ConvertTo-EnvMap $Lines + $entries = [ordered]@{} + foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) } + [pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) } +} + +# List of differences (names only). Empty when identical. +function Compare-EnvFingerprint($Before, $After) { + $d = New-Object System.Collections.Generic.List[string] + foreach ($k in $Before.Entries.Keys) { + if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") } + elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") } + } + foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } } + if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") } + , $d.ToArray() +} + +function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') } + +# Re-read the env block and abort on any difference from the preflight fingerprint. +function Assert-EnvUnchanged($ctx, [string]$When) { + $now = Get-EnvFingerprint (Read-EnvBlockLines) + $diff = Compare-EnvFingerprint $ctx.EnvFp $now + if ($diff.Count -gt 0) { + # During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead. + if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return } + Fail "NSSM env block changed ${When}: $($diff -join '; ')" + } + Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))" +} + +# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call +# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine. +function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) { + if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return } + $o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly } + try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" } +} +function Get-Dirs([string]$Dir, [string]$Filter = '*') { + if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return } + try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" } +} + +function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } } + +function Get-DirSize([string]$p) { + if (-not (Test-Path -LiteralPath $p)) { return 0 } + $s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum + if ($s) { [double]$s } else { 0 } +} + +# Version of a self-contained .NET runtime in a directory (from mscordaccore___.dll), or $null (framework-dependent). +function Get-BundledRuntimeVersion([string]$Dir) { + $f = @(Get-Files $Dir 'mscordaccore_*.dll') + foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } } + $null +} + +# --- processes -------------------------------------------------------------------------------------------------- + +function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) } +function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) } + +function Get-GatewayProcesses($ctx) { + @(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object { + $role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' } + $_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force + }) +} +function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' } + +function Assert-NoGatewayProcess($ctx, [string]$When) { + $left = @(Get-GatewayProcesses $ctx) + if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" } +} + +# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort. +function Assert-SingleGateway($ctx, [string]$State) { + $procs = @(Get-GatewayProcesses $ctx) + $servers = @($procs | Where-Object { $_.Role -eq 'Server' }) + $workers = @($procs | Where-Object { $_.Role -eq 'Worker' }) + Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })" + if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." } + switch ($State) { + 'SERVICE_STOPPED' { + if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." } + } + 'SERVICE_RUNNING' { + if ($servers.Count -eq 0) { + Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing." + } + $s = $servers[0] + if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" } + if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" } + $hostPid = Get-ServiceHostPid + if ($hostPid) { + if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" } + Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)" + } else { Warn 'service host PID unavailable; parent check skipped' } + $foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id }) + if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." } + $wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) }) + if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" } + } + 'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' } + default { + if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" } + } + } +} + +# --- context discovery (read only) ------------------------------------------------------------------------- + +function Get-Context { + $state = Get-ServiceState + if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" } + $ctx = [ordered]@{} + $ctx.State = $state + $ctx.Application = Get-NssmValue 'Application' + $ctx.AppDirectory = Get-NssmValue 'AppDirectory' + $lines = @(Read-EnvBlockLines) + $ctx.Env = ConvertTo-EnvMap $lines + $ctx.EnvFp = Get-EnvFingerprint $lines + $ctx.EnvCount = $ctx.EnvFp.Count + if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory } + $wexe = $ctx.Env[$script:EnvWorkerExe] + $ctx.WorkerExe = $wexe + if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir } + elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe } + else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" } + $pd = 'C:\ProgramData\MxGateway' + $auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" } + $sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" } + $ctx.AuthDb = $auth; $ctx.SecretsDb = $sec + $ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath'] + $ctx.ServerExeName = Get-Leaf $ctx.Application + $ctx.WorkerExeName = Get-Leaf $wexe + if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' } + # Process image names: the installed ones first, then both known schemes. + $script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique) + $script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique) + [pscustomobject]$ctx +} + +# --- preflight ----------------------------------------------------------------------------------------------- + +function Test-MasterKey($ctx) { + $src = $ctx.Env['Secrets__MasterKey__Source'] + $envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' } + if ($ctx.KekFile -or $src -eq 'File') { + if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' } + if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" } + foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } } + if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" } + } elseif ($ctx.Env.Contains($envVar)) { + Info "master key: env block entry $envVar" + } else { + Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start." + } +} + +function Invoke-Preflight($ctx) { + Say '== Preflight ==' + Info "service '$ServiceName' state: $($ctx.State)" + Info "Application : $($ctx.Application)" + Info "AppDirectory: $($ctx.AppDirectory)" + Info "Server dir : $($ctx.ServerDir)" + Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))" + Info "API-key DB : $($ctx.AuthDb)" + Info "Secrets DB : $($ctx.SecretsDb)" + Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)" + + if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' } + if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) { + Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify' + } + if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') { + Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade' + } + if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" } + if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" } + + # State that must survive: rooted and outside the swapped directories. + foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) { + if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" } + foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { + if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." } + } + if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" } + else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" } + } + Test-MasterKey $ctx + + # Directories + foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) { + if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" } + } + if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) { + Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories' + } + if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" } + if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" } + $dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' }) + if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" } + + # One-gateway guard (fail closed) + Assert-SingleGateway $ctx $ctx.State + $others = Get-OtherGatewayServices $ctx.ServerExeName + if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' } + elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" } + else { Info "no other service launches $($ctx.ServerExeName)" } + + if ($Rollback) { Say '== Preflight OK =='; return } + + # Sources + if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' } + $stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName + $stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName + foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) { + if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" } + } + if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' } + $arch = Get-ExeArchitecture $stagedWorkerExe + if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" } + Info "staged Worker architecture: X86" + $sSha = Get-ProductSha $stagedServerExe + $wSha = Get-ProductSha $stagedWorkerExe + Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)" + if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" } + if ($ExpectedSha) { + if (-not (Test-ShaMatch $sSha $ExpectedSha)) { + if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" } + else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" } + } + } else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' } + $leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null } + if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" } + + # Runtime: never silently downgrade a self-contained Server. + $liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource + Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })" + if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' } + if ($liveRt -and $newRt -and $newRt -lt $liveRt) { + if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" } + else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" } + } + + # What the swap will carry over / drop (top-level files). + foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) { + $plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings) + Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })" + if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" } + } + $lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name }) + $onlyNew = @($nw | Where-Object { $lw -notcontains $_ }) + if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" } + + # Disk space: 2x (server + worker + stores + payload), at least MinFreeGB + if (-not $SkipFreeSpaceCheck) { + $need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource) + foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length } + $need = [math]::Max($need * 2, $MinFreeGB * 1GB) + $root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path) + $free = (New-Object IO.DriveInfo $root).AvailableFreeSpace + Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB)) + if ($free -lt $need) { Fail 'not enough free disk space for the backups' } + } + Say '== Preflight OK ==' +} + +# --- stop / start --------------------------------------------------------------------------------------------- + +function Wait-State([string]$Want, [int]$TimeoutSec) { + $deadline = (Get-Date).AddSeconds($TimeoutSec) + do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline) + $false +} + +function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) { + $deadline = (Get-Date).AddSeconds($TimeoutSec) + do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline) + $false +} + +# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name. +function Stop-InstallProcesses($ctx, [string]$Why) { + $installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ } + $mine = @(Get-GatewayProcesses $ctx | Where-Object { + if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name } + } | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } }) + foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id } + $mine.Count +} + +function Stop-Gateway($ctx) { + Say '-- Stop --' + $state = Get-ServiceState + if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' } + else { + Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" { + [void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail) + if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) { + # STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers. + Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes" + $n = Stop-InstallProcesses $ctx 'STOP_PENDING' + if ($n -eq 0) { Warn 'no process of this install was found to kill' } + if (-not (Wait-State 'SERVICE_STOPPED' 20)) { + Warn 'still not STOPPED; killing the NSSM service host process' + $svcPid = Get-ServiceHostPid + if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid } + if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" } + } + } + if (-not (Wait-NoGatewayProcess $ctx 20)) { + Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))" + [void](Stop-InstallProcesses $ctx 'left over after the stop') + [void](Wait-NoGatewayProcess $ctx 10) + } + } + } + if (-not $script:Dry) { + Assert-NoGatewayProcess $ctx 'after stop' + Info 'stopped; no gateway/Worker process on the box' + Assert-EnvUnchanged $ctx 'after the stop' + } +} + +function Start-Gateway($ctx) { + Say '-- Start --' + if (-not $script:Dry) { + Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)' + Assert-EnvUnchanged $ctx 'before the start' + } else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" } + Step "nssm start $ServiceName (once)" { + # nssm exits non-zero with 'Unexpected status SERVICE_START_PENDING' when the app is slow to report RUNNING; + # under ErrorActionPreference=Stop its stderr line throws. The RUNNING wait below is the real gate. + try { $null = & $NssmPath start $ServiceName 2>&1 } catch { Say "[info] nssm start: $($_.Exception.Message) -- deciding by the RUNNING wait below" } + } + Step "wait for SERVICE_RUNNING and compare the env block fingerprint" { + if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" } + Assert-EnvUnchanged $ctx 'after the start' + } +} + +# --- backup / swap ----------------------------------------------------------------------------------------------- + +function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } } + +function Backup-State($ctx) { + Say '-- Backup (service stopped) --' + $suffix = Get-Suffix + $root = Split-Path -Parent $ctx.ServerDir + $dir = Join-Path $root "_deploy-backup-$suffix" + $script:BackupDir = $dir + Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" { + New-Item -ItemType Directory -Path $dir -Force | Out-Null + Protect-Directory $dir + $body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines) + Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8 + Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg") + $fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" }) + Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII + } + foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) { + foreach ($f in (Get-StoreFiles $db)) { + $dst = "$f.bak-$suffix" + Step "copy store file $f -> $dst (live file is never moved)" { + Copy-Item -LiteralPath $f -Destination $dst -Force + if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" } + } + } + if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" } + } + if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) { + Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force } + } +} + +function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false } + +# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins). +function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) { + $liveFiles = @(Get-Files $Live) + $srcNames = @(Get-Files $Source | ForEach-Object { $_.Name }) + $srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name }) + $carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string] + foreach ($f in $liveFiles) { + $inSource = $srcNames -contains $f.Name + if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue } + if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue } + if ($inSource) { continue } + if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) } + } + $dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name }) + [pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs } +} + +function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) { + $bak = "$Live.bak-$(Get-Suffix)" + $plan = Get-CarryPlan $Live $Source $KeepSettings + $hashes = @{} + foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) } + $sddlOld = Get-DirSddl $Live + Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) } + Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force } + foreach ($f in $plan.Carry) { + Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" { + Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force + if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" } + Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f) + } + } + foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" } + if (-not $script:Dry) { + $sddlNew = Get-DirSddl $Live + if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" } + } + $bak +} + +# --- verification ---------------------------------------------------------------------------------------------------- + +function Invoke-Verify($ctx, [bool]$AfterRollback = $false) { + Say '== Verify ==' + if ($script:Dry) { + Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)" + Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })" + Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged" + return + } + $ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds) + while (-not $ready -and (Get-Date) -lt $deadline) { + $ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200) + if (-not $ready) { Start-Sleep -Seconds 3 } + } + if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" } + Info '/health/ready = 200' + $hz = Get-HttpStatus "$HealthBaseUrl/healthz" + if ($hz -ne 200) { Fail "/healthz = $hz" } + Info '/healthz = 200' + + $sha = Get-ProductSha $ctx.Application + Info "deployed Server SHA (ProductVersion): $sha" + if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" } + + Assert-SingleGateway $ctx 'SERVICE_RUNNING' + $pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id + Info "watching the service for $StableSeconds s (gateway #$pid0)" + $until = (Get-Date).AddSeconds($StableSeconds) + do { + if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' } + Start-Sleep -Seconds 5 + } while ((Get-Date) -lt $until) + $servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' }) + if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" } + Assert-EnvUnchanged $ctx 'after the stability window' + Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)' +} + +# --- rollback ------------------------------------------------------------------------------------------------------------ + +# Backup dirs come in two spellings: .bak-yyyyMMdd-HHmmss[-label] (this script) and .bak.yyyyMMddTHHmmss (manual / adapted scripts). +function Get-BackupKey([string]$Name) { + if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" } + $null +} + +function Find-Backup([string]$Live, [string]$Stamp) { + $leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live + $cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) }) + if ($Stamp) { + $key = ($Stamp -replace '[^0-9]', '') + $cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" }) + } + $cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1 +} + +function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) { + Say '== Rollback ==' + $sb = Find-Backup $ctx.ServerDir $Stamp + if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" } + $key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') } + $wb = Find-Backup $ctx.WorkerDir $key + if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null } + if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" } + if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" } + Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })" + $fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ }) + if ($fpFile.Count) { + $saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' } + $now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" }) + if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" } + else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" } + } + Stop-Gateway $ctx + foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) { + if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue } + $failed = "$($pair[0]).failed-$($script:Stamp)" + if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } } + Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) } + } + if ($Stores) { + $bstamp = ($sb.Name -replace '^.*\.bak[-.]', '') + foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) { + foreach ($ext in '', '-wal', '-shm') { + $live = "$db$ext"; $bak = "$db$ext.bak-$bstamp" + if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } } + elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) { + # a stale wal/shm must not pair with the restored db: move it aside, never delete + Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") } + } + elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" } + } + } + } else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' } + Start-Gateway $ctx +} + +# --- main ------------------------------------------------------------------------------------------------------------------- + +function Invoke-Main { + $exit = 0 + $ctx = $null + try { + Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME" + $ctx = Get-Context + Invoke-Preflight $ctx + + if (-not $script:Dry) { + $lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock' + try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') } + catch { Fail "another deploy run holds $lockPath" } + } + + if ($Rollback) { + Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false + Invoke-Verify $ctx + } else { + Stop-Gateway $ctx + try { Backup-State $ctx } + catch { + $err = $_ + if (-not $script:Dry) { + Warn 'backup failed before anything was swapped - starting the unchanged build again' + try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" } + } + throw $err + } + $touched = $false + try { + $touched = $true + $null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings) + $null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false + if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' } + Start-Gateway $ctx + Invoke-Verify $ctx + } catch { + $err = $_ + Say "[FAIL] $(Hide $err.Exception.Message)" + if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw } + Warn 'upgrade failed - rolling back automatically' + $script:EnvWarnOnly = $true + try { + Invoke-RollbackSwap $ctx $script:Stamp $false $true + Invoke-Verify $ctx $true + Warn 'ROLLED BACK to the previous build; original failure follows' + } catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." } + throw $err + } + } + Say '== DONE ==' + } catch { + $exit = 1 + $m = Hide $_.Exception.Message + $script:Notes.Add("ERROR: $m") + Write-Host "ERROR: $m" -ForegroundColor Red + } finally { + if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null } + try { + if (-not $script:Dry -and $ctx) { + $notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt" + Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8 + Write-Host "deploy notes: $notes" + } + } catch { } + $script:Redact.Clear() + } + $exit +} + +# Dot-sourcing (the dry-run harness) only defines the functions. +if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc } diff --git a/docs/deploy/2026-11-19-prod-stage/mxgw/appsettings.json.prod b/docs/deploy/2026-11-19-prod-stage/mxgw/appsettings.json.prod new file mode 100644 index 0000000..3ee732e --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/mxgw/appsettings.json.prod @@ -0,0 +1,117 @@ +{ + "Serilog": { + "Using": [ + "Serilog.Sinks.Console", + "Serilog.Sinks.File" + ], + "MinimumLevel": { + "Default": "Information", + "Override": { + "Microsoft.AspNetCore": "Warning" + } + }, + "WriteTo": [ + { + "Name": "Console" + }, + { + "Name": "File", + "Args": { + "path": "logs/mxgateway-.log", + "rollingInterval": "Day", + "retainedFileCountLimit": 30, + "fileSizeLimitBytes": 52428800, + "rollOnFileSizeLimit": true + } + } + ] + }, + "AllowedHosts": "*", + "Secrets": { + "MasterKey": { + "Source": "Environment", + "EnvVarName": "ZB_SECRETS_MASTER_KEY" + }, + "RunMigrationsOnStartup": true, + "ResolveCacheTtl": "00:00:30" + }, + "MxGateway": { + "Authentication": { + "Mode": "ApiKey", + "PepperSecretName": "MxGateway:ApiKeyPepper", + "RunMigrationsOnStartup": true + }, + "Ldap": { + "Enabled": true, + "Server": "localhost", + "Port": 3893, + "Transport": "None", + "AllowInsecure": true, + "SearchBase": "dc=zb,dc=local", + "ServiceAccountDn": "cn=serviceaccount,dc=zb,dc=local", + "ServiceAccountPassword": "${secret:ldap/mxgateway/bind}", + "UserNameAttribute": "cn", + "DisplayNameAttribute": "cn", + "GroupAttribute": "memberOf" + }, + "Worker": { + "ExecutablePath": "src\\ZB.MOM.WW.MxGateway.Worker\\bin\\x86\\Release\\ZB.MOM.WW.MxGateway.Worker.exe", + "RequiredArchitecture": "X86", + "StartupTimeoutSeconds": 30, + "ShutdownTimeoutSeconds": 10, + "HeartbeatCheckIntervalSeconds": 5, + "HeartbeatGraceSeconds": 15, + "HeartbeatStuckCeilingSeconds": 75, + "WorkerHeartbeatSendIntervalSeconds": 5, + "WorkerHeartbeatGraceSeconds": 15, + "WorkerHeartbeatStuckCeilingSeconds": 75, + "EventChannelFullModeTimeoutMilliseconds": 5000, + "MaxMessageBytes": 16842752 + }, + "Sessions": { + "DefaultCommandTimeoutSeconds": 30, + "MaxSessions": 64, + "MaxPendingCommandsPerSession": 128, + "DefaultLeaseSeconds": 1800, + "LeaseSweepIntervalSeconds": 30, + "DetachGraceSeconds": 30, + "UnattachedSessionGraceSeconds": 0, + "AllowMultipleEventSubscribers": false, + "MaxEventSubscribersPerSession": 8 + }, + "Events": { + "QueueCapacity": 10000, + "BackpressurePolicy": "FailFast", + "ReplayBufferCapacity": 1024, + "ReplayRetentionSeconds": 300, + "ReplayBufferMaxBytes": 67108864 + }, + "Dashboard": { + "Enabled": true, + "AllowAnonymousLocalhost": true, + "SnapshotIntervalMilliseconds": 1000, + "RecentFaultLimit": 100, + "RecentSessionLimit": 200, + "ShowTagValues": false + }, + "Protocol": { + "WorkerProtocolVersion": 1, + "MaxGrpcMessageBytes": 16777216 + }, + "Galaxy": { + "ConnectionString": "Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;", + "CommandTimeoutSeconds": 60, + "DashboardRefreshIntervalSeconds": 30, + "PersistSnapshot": true + }, + "Alarms": { + "Enabled": true, + "SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV", + "DefaultArea": "", + "ReconcileIntervalSeconds": 30, + "PollIntervalMilliseconds": 500, + "MaxAlarmsPerFetch": 1024, + "AckClientName": "MxGateway" + } + } +} diff --git a/docs/deploy/2026-11-19-prod-stage/mxgw/install-mxgw.ps1 b/docs/deploy/2026-11-19-prod-stage/mxgw/install-mxgw.ps1 new file mode 100644 index 0000000..f3ae17d --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/mxgw/install-mxgw.ps1 @@ -0,0 +1,99 @@ +# PROD MxAccessGateway first install (plan Phase 4 + S3A). Run ON the box, only after the owner's go for Phase 4. +# -Mode dry : verify the zip, the MXAccess gate and the env, expand into .\dry\, run Install-MxGatewayService -DryRun. +# Creates nothing outside this stage folder. +# -Mode real : expand into E:\ApiInstall\MxAccessGateway\{app,worker}, put appsettings.json.prod in place, create the +# KEK (or take -KekFrom), install the NSSM service STOPPED (never -Start: seed the LDAP secret first, S3A.4). +# -ValuesDir holds one .txt per '@file' line in mxgw.env (pepper, Galaxy connection string). It is read and never +# printed; the caller deletes it afterwards. Secret values never go on argv. +param( + [Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode, + [string]$ValuesDir, + [string]$KekFrom +) +$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue' +$stage = $PSScriptRoot +$zip = Join-Path $stage 'mxgw-e74918cc.zip' +$zipSha = 'F0999B75BEE196375AEB8E3E277A0E948C54A747FD5251D336461C9665D847F2' +$root = 'E:\ApiInstall\MxAccessGateway' +$kekDir = 'C:\ProgramData\ZB\kek' +$kek = "$kekDir\mxgateway.kek" +$nssm = 'E:\psrem-stage\prod-common\nssm.exe' +$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } +function Fail($m) { throw "ABORT: $m" } + +Say "install-mxgw mode=$Mode box=$env:COMPUTERNAME" +# 1. MXAccess gate (plan Q3): the PAR boxes have none until WinPlatform is deployed from the prod Galaxy. +if (-not (Test-Path -LiteralPath $lmx)) { Fail "no $lmx - MXAccess not installed on this box (plan Q3 gate)" } +Say "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12))" +# 2. artifact +$h = (Get-FileHash $zip -Algorithm SHA256).Hash +if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" } +Say 'zip sha256 ok' +if (-not (Test-Path $nssm)) { Fail "nssm not staged at $nssm" } +# 3. env: placeholders + @file values +$lines = Get-Content (Join-Path $stage 'mxgw.env') +$left = @($lines | Where-Object { $_ -notmatch '^\s*#' -and $_ -match '__Q\d+_' }) +if ($left.Count) { Fail "unresolved placeholders in mxgw.env: $(($left | ForEach-Object { ($_ -split '=')[0] }) -join ', ')" } +$out = New-Object System.Collections.Generic.List[string] +foreach ($l in $lines) { + if ($l -match '^\s*(#|$)') { continue } + $k, $v = $l -split '=', 2 + if ($v -eq '@file') { + if (-not $ValuesDir) { Fail "$k needs -ValuesDir" } + $f = Join-Path $ValuesDir "$k.txt" + if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" } + $v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n") + if (-not $v) { Fail "empty value file for $k" } + } + $out.Add("$k=$v") +} +Say "env entries: $($out.Count)" +if ($out | Where-Object { $_ -match '_(DEV|QA)=' }) { Fail 'a _DEV/_QA group name is in a PROD env (plan S3A.2)' } + +# 4. layout +if ($Mode -eq 'dry') { $base = Join-Path $stage 'dry' } else { $base = $root } +$app = Join-Path $base 'app'; $worker = Join-Path $base 'worker' +if ($Mode -eq 'real') { + if ((Test-Path $app) -or (Test-Path $worker)) { Fail "$app or $worker already exists - this is the first-install script; use Upgrade-MxGatewayService.ps1" } + if (Get-Service MxAccessGw -ErrorAction SilentlyContinue) { Fail 'service MxAccessGw already exists' } +} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force } +$tmp = Join-Path $stage "expand-$(Get-Date -Format yyyyMMddTHHmmss)" +Expand-Archive -LiteralPath $zip -DestinationPath $tmp +New-Item -ItemType Directory -Force $base | Out-Null +Move-Item (Join-Path $tmp 'Server') $app +Move-Item (Join-Path $tmp 'Worker') $worker +Remove-Item $tmp -Recurse -Force +Copy-Item (Join-Path $stage 'appsettings.json.prod') (Join-Path $app 'appsettings.json') -Force +if (Select-String -Path (Join-Path $app 'appsettings.json') -Pattern 'GwAdmin|GwReader' -Quiet) { Fail 'GwAdmin/GwReader still in appsettings.json' } +Say "expanded to $base (app: $(@(Get-ChildItem $app -Recurse -File).Count) files, worker: $(@(Get-ChildItem $worker -Recurse -File).Count) files)" +foreach ($e in (Join-Path $app 'ZB.MOM.WW.MxGateway.Server.exe'), (Join-Path $worker 'MxGateway.Worker.exe')) { + Say "$(Split-Path $e -Leaf) $((Get-Item $e).VersionInfo.ProductVersion)" } + +# 5. KEK (32 raw random bytes; Administrators + SYSTEM only) +if ($Mode -eq 'real') { + New-Item -ItemType Directory -Force $kekDir | Out-Null + & icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null + if (Test-Path $kek) { Say "KEK exists, kept: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" } + elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say "KEK copied from -KekFrom: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" } + else { + $b = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b) + [IO.File]::WriteAllBytes($kek, $b); Say "KEK generated: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" + } + if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' } +} + +# 6. service (stopped). Env goes through a protected temp file that is always deleted. +$envTmp = Join-Path $stage "_env-$PID.tmp" +try { + [IO.File]::WriteAllLines($envTmp, $out) + & icacls.exe $envTmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null + $a = @{ ServerDir = $app; WorkerDir = $worker; NssmPath = $nssm; DataDir = "$root\data"; LogDir = "$root\logs" + EnvFile = $envTmp; WorkerExeName = 'MxGateway.Worker.exe' } + if ($Mode -eq 'dry') { $a.DryRun = $true } + & (Join-Path $stage 'Install-MxGatewayService.ps1') @a + $rc = $LASTEXITCODE +} finally { Remove-Item $envTmp -Force -ErrorAction SilentlyContinue } +if ($rc -ne 0) { Fail "Install-MxGatewayService exited $rc" } +if ($Mode -eq 'real') { Say 'INSTALLED STOPPED. Next: seed nam-ldap-svc-password (prod-common\seed-secret-prod.ps1), review the env, then start (plan S3A.4/S3A.5).' } +Say "DONE mode=$Mode" diff --git a/docs/deploy/2026-11-19-prod-stage/mxgw/mxgw.env b/docs/deploy/2026-11-19-prod-stage/mxgw/mxgw.env new file mode 100644 index 0000000..2666f63 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/mxgw/mxgw.env @@ -0,0 +1,42 @@ +# MxAccessGateway NSSM env for PROD (all four boxes; QA vq03 key set + PROD plan §3A.3). KEY=VALUE, '#' comments. +# @file = value read at install time from \.txt (secrets; never in this file) +# __Q2_..__ = open question Q2 (prod Galaxy Repository / alarm area); install-mxgw.ps1 refuses while any remain +ASPNETCORE_ENVIRONMENT=Production +DOTNET_ENVIRONMENT=Production +Kestrel__Endpoints__Grpc__Url=http://0.0.0.0:5120 +Kestrel__Endpoints__Grpc__Protocols=Http2 +Kestrel__Endpoints__Dashboard__Url=http://0.0.0.0:5130 +Kestrel__Endpoints__Dashboard__Protocols=Http1 +MxGateway__Worker__ExecutablePath=E:\ApiInstall\MxAccessGateway\worker\MxGateway.Worker.exe +MxGateway__Worker__RequiredArchitecture=X86 +MxGateway__Worker__WorkingDirectory=E:\ApiInstall\MxAccessGateway\worker +MxGateway__Worker__MaxMessageBytes=16842752 +MxGateway__Protocol__MaxGrpcMessageBytes=16777216 +MxGateway__Galaxy__ConnectionString=@file +MxGateway__Authentication__Mode=ApiKey +MxGateway__Authentication__SqlitePath=E:\ApiInstall\MxAccessGateway\data\gateway-auth.db +MxGateway__ApiKeyPepper=@file +MxGateway__Alarms__Enabled=true +MxGateway__Alarms__SubscriptionExpression=__Q2_ALARM_EXPRESSION__ +MxGateway__Ldap__Enabled=true +MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com +MxGateway__Ldap__Port=636 +MxGateway__Ldap__Transport=Ldaps +MxGateway__Ldap__AllowInsecure=false +MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636 +MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com +MxGateway__Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com +MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password} +MxGateway__Ldap__UserNameAttribute=sAMAccountName +MxGateway__Ldap__DisplayNameAttribute=name +MxGateway__Ldap__GroupAttribute=memberOf +MxGateway__Ldap__BindMode=ServiceAccount +MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_PROD=Administrator +MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_PROD=Viewer +MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_PROD=Viewer +MxGateway__Dashboard__RequireHttpsCookie=false +Secrets__SqlitePath=E:\ApiInstall\MxAccessGateway\data\secrets.db +Secrets__MasterKey__Source=File +Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\mxgateway.kek +MxGateway__Telemetry__Exporter=Otlp +MxGateway__Telemetry__OtlpEndpoint=http://127.0.0.1:4317 diff --git a/docs/deploy/2026-11-19-prod-stage/otel/Install-OtelCollector.ps1 b/docs/deploy/2026-11-19-prod-stage/otel/Install-OtelCollector.ps1 new file mode 100644 index 0000000..f4d018a --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/otel/Install-OtelCollector.ps1 @@ -0,0 +1,116 @@ +<# +.SYNOPSIS + Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54). + +.DESCRIPTION + Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\} + Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit. + ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit + ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and + Modify on data\ and logs\. + Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service + wrapper, so otelcol runs in console mode). + Every listener in config.yaml binds 127.0.0.1. The script checks that after start. + +.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt. +.PARAMETER Tier DEV, QA or PROD (PROD added in the 2026-11-19 PROD stage copy). +.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\. +.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed. (kept). +#> +param( + [Parameter(Mandatory)] [string]$StageDir, + [ValidateSet('DEV','QA','PROD')] [string]$Tier, + [string]$NssmPath, + [switch]$Uninstall +) +$ErrorActionPreference = 'Stop' +$svcName = 'ZbOtelCollector' +$root = 'E:\ApiInstall\OtelCollector' +$acct = "NT SERVICE\$svcName" +$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } + +if ($Uninstall) { + $s = Get-Service $svcName -ErrorAction SilentlyContinue + if ($s) { + if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) } + & sc.exe delete $svcName | Out-Null + Say "service $svcName deleted" + } + if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" } + return +} + +if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' } +if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" } +if (Test-Path $root) { throw "$root already exists" } + +# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git). +$sums = @{} +Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } | + ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() } +foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') { + $h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash + if ($h -ne $sums[$f]) { throw "hash mismatch $f" } + Say "hash ok $f $($h.Substring(0,12))" +} + +# 2. Layout +foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null } +$tmp = Join-Path $StageDir 'x' +if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force } +Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp +$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash +if ($h -ne $exeSha) { throw "exe hash mismatch $h" } +Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe') +Remove-Item $tmp -Recurse -Force +Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe') +Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml') +Say "layout ok, exe sha $($exeSha.Substring(0,12))" + +# 3. Service +$nssm = Join-Path $root 'bin\nssm.exe' +& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null +$set = @( + @('AppParameters', "--config=$root\config\config.yaml"), + @('AppDirectory', $root), + @('DisplayName', 'ZB OpenTelemetry Collector'), + @('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'), + @('Start', 'SERVICE_AUTO_START'), + @('AppStdout', "$root\logs\collector.log"), + @('AppStderr', "$root\logs\collector.log"), + @('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'), + @('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000')) +foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } } +Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1') +& sc.exe config $svcName obj= $acct | Out-Null +if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" } +Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)" + +# 4. ACL +& icacls $root /inheritance:d | Out-Null +& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null +& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null +foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null } +(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() } + +# 5. Start + checks +Start-Service $svcName +$ok = $false +for ($i = 0; $i -lt 30; $i++) { + Start-Sleep 2 + try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {} +} +Say "service $((Get-Service $svcName).Status); health_check ok=$ok" +if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' } +$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId +$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe' +$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId +$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" } +$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' } +if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" } +$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces' +Say "OTLP/HTTP empty POST -> $($r.StatusCode)" +$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false +Say "collector.log error lines: $(@($err).Count)" +Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) } diff --git a/docs/deploy/2026-11-19-prod-stage/otel/Test-OtelCollector.ps1 b/docs/deploy/2026-11-19-prod-stage/otel/Test-OtelCollector.ps1 new file mode 100644 index 0000000..913fc6b --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/otel/Test-OtelCollector.ps1 @@ -0,0 +1,65 @@ +<# +.SYNOPSIS + Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent. +.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window. +.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP). +#> +param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20) +$ErrorActionPreference = 'Continue' +$root = 'E:\ApiInstall\OtelCollector' +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } +$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'" +Say "service $($s.State) $($s.StartMode) $($s.StartName)" +try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" } +$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe' +if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" } + " collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) } +Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime } +" E: free GB " + [int]((Get-PSDrive E).Free/1GB) + +$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000 +function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue } +foreach ($sig in 'metrics','traces','logs') { + $f = "$root\data\$sig.jsonl" + if (-not (Test-Path $f)) { Say "${sig}: no file"; continue } + $agg = @{} + foreach ($line in (Get-Content $f -Tail $Tail)) { + try { $o = $line | ConvertFrom-Json } catch { continue } + $rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs } + foreach ($r in $rs) { + $key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role') + if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } } + $a = $agg[$key] + if ($sig -eq 'metrics') { + foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1 + foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } } + } elseif ($sig -eq 'traces') { + foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } } + } else { + foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } } + } + } + } + Say "$sig (last $Tail export requests):" + foreach ($k in ($agg.Keys | Sort-Object)) { + $a = $agg[$k] + $newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' } + $fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' } + " $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh" + if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') } + } +} +try { + $p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n" + $jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name + Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" } +} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" } +try { + $q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n" + Say "collector self-metrics:" + $q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') } +} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" } +foreach ($port in $AppPorts) { + $c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} } + " app :$port /metrics $c" +} diff --git a/docs/deploy/2026-11-19-prod-stage/otel/config.yaml b/docs/deploy/2026-11-19-prod-stage/otel/config.yaml new file mode 100644 index 0000000..3382867 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/otel/config.yaml @@ -0,0 +1,80 @@ +# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54). +# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA), +# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector. +# +# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317, +# and nothing is reachable from the network. That is why the receivers have no auth and no TLS. +# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md). +# +# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as +# deployment.environment.name, so files copied off two tiers can still be told apart. +# +# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older +# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB. + +extensions: + health_check: + endpoint: 127.0.0.1:13133 + +receivers: + otlp: + protocols: + grpc: + endpoint: 127.0.0.1:4317 + http: + endpoint: 127.0.0.1:4318 + +processors: + memory_limiter: + check_interval: 5s + limit_mib: 256 + spike_limit_mib: 64 + resource/tier: + attributes: + - key: deployment.environment.name + value: ${env:ZB_TIER} + action: upsert + batch: {} + +exporters: + file/metrics: + path: E:/ApiInstall/OtelCollector/data/metrics.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + file/traces: + path: E:/ApiInstall/OtelCollector/data/traces.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + file/logs: + path: E:/ApiInstall/OtelCollector/data/logs.jsonl + rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 } + prometheus: + endpoint: 127.0.0.1:8889 + resource_constant_labels: + included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name] + metric_expiration: 15m + +service: + extensions: [health_check] + telemetry: + logs: + level: info + metrics: + level: basic + readers: + - pull: + exporter: + prometheus: + host: 127.0.0.1 + port: 8888 + pipelines: + metrics: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/metrics, prometheus] + traces: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/traces] + logs: + receivers: [otlp] + processors: [memory_limiter, resource/tier, batch] + exporters: [file/logs] diff --git a/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp03 b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp03 new file mode 100644 index 0000000..4198f91 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp03 @@ -0,0 +1,113 @@ +{ + "ScadaBridge": { + "Node": { + "Role": "Central", + "NodeName": "central-a", + "NodeHostname": "10.220.153.205", + "RemotingPort": 8081 + }, + "_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.", + "Cluster": { + "SeedNodes": [ + "akka.tcp://scadabridge@10.220.153.205:8081", + "akka.tcp://scadabridge@10.220.153.210:8081" + ], + "SplitBrainResolverStrategy": "leader-auto-down", + "StableAfter": "00:00:15", + "HeartbeatInterval": "00:00:02", + "FailureDetectionThreshold": "00:00:10", + "MinNrOfMembers": 1, + "BootstrapGuard": { + "Enabled": true + } + }, + "Database": { + "ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}" + }, + "_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.", + "Security": { + "Ldap": { + "Enabled": true, + "Server": "uswap-vdc-nam1.nam.zimmer.com", + "Port": 636, + "Transport": "Ldaps", + "AllowInsecure": false, + "FallbackServers": [ + "uswap-vdc-nam2.nam.zimmer.com:636" + ], + "SearchBase": "DC=nam,DC=zimmer,DC=com", + "ServiceAccountDn": "svc_igntadmin@nam.zimmer.com", + "ServiceAccountPassword": "${secret:nam-ldap-svc-password}", + "UserNameAttribute": "sAMAccountName", + "DisplayNameAttribute": "name", + "GroupAttribute": "memberOf", + "BindMode": "ServiceAccount" + }, + "JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}", + "JwtExpiryMinutes": 25, + "IdleTimeoutMinutes": 60, + "RequireHttpsCookie": false, + "ForwardedHeaders": { + "Enabled": false + } + }, + "_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).", + "Communication": { + "DeploymentTimeout": "00:02:00", + "LifecycleTimeout": "00:00:30", + "QueryTimeout": "00:00:30", + "TransportHeartbeatInterval": "00:00:05", + "TransportFailureThreshold": "00:00:15", + "CentralFetchBaseUrl": "http://10.220.153.205:8085" + }, + "ManagementService": { + "CommandTimeout": "00:00:30", + "LongRunningCommandTimeout": "00:05:00" + }, + "HealthMonitoring": { + "ReportInterval": "00:00:30", + "OfflineTimeout": "00:01:00" + }, + "InboundApi": { + "DefaultMethodTimeout": "00:00:30", + "ApiKeyStore": { + "Provider": "SqlServer" + }, + "StandbyForwardPeerEndpoints": [ + "http://10.220.153.210:8085" + ] + }, + "Notification": { + "SmtpServer": "REPLACE_WITH_SMTP_HOST", + "SmtpPort": 25, + "AuthMode": "None", + "FromAddress": "scada-notifications@zimmer.com" + }, + "NotificationOutbox": { + "DispatchInterval": "00:00:10", + "DispatchBatchSize": 100, + "StuckAgeThreshold": "00:10:00", + "TerminalRetention": "365.00:00:00", + "PurgeInterval": "1.00:00:00", + "DeliveredKpiWindow": "00:01:00" + }, + "Logging": { + "MinimumLevel": "Information" + } + }, + "Secrets": { + "Store": { + "Provider": "SqlServer" + } + }, + "Serilog": { + "WriteTo": [ + {}, + { + "Args": { + "path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log" + } + } + ] + } +} \ No newline at end of file diff --git a/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp04 b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp04 new file mode 100644 index 0000000..4180f44 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Central.json.prod-vp04 @@ -0,0 +1,113 @@ +{ + "ScadaBridge": { + "Node": { + "Role": "Central", + "NodeName": "central-b", + "NodeHostname": "10.220.153.210", + "RemotingPort": 8081 + }, + "_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.", + "Cluster": { + "SeedNodes": [ + "akka.tcp://scadabridge@10.220.153.210:8081", + "akka.tcp://scadabridge@10.220.153.205:8081" + ], + "SplitBrainResolverStrategy": "leader-auto-down", + "StableAfter": "00:00:15", + "HeartbeatInterval": "00:00:02", + "FailureDetectionThreshold": "00:00:10", + "MinNrOfMembers": 1, + "BootstrapGuard": { + "Enabled": true + } + }, + "Database": { + "ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}" + }, + "_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.", + "Security": { + "Ldap": { + "Enabled": true, + "Server": "uswap-vdc-nam1.nam.zimmer.com", + "Port": 636, + "Transport": "Ldaps", + "AllowInsecure": false, + "FallbackServers": [ + "uswap-vdc-nam2.nam.zimmer.com:636" + ], + "SearchBase": "DC=nam,DC=zimmer,DC=com", + "ServiceAccountDn": "svc_igntadmin@nam.zimmer.com", + "ServiceAccountPassword": "${secret:nam-ldap-svc-password}", + "UserNameAttribute": "sAMAccountName", + "DisplayNameAttribute": "name", + "GroupAttribute": "memberOf", + "BindMode": "ServiceAccount" + }, + "JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}", + "JwtExpiryMinutes": 25, + "IdleTimeoutMinutes": 60, + "RequireHttpsCookie": false, + "ForwardedHeaders": { + "Enabled": false + } + }, + "_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).", + "Communication": { + "DeploymentTimeout": "00:02:00", + "LifecycleTimeout": "00:00:30", + "QueryTimeout": "00:00:30", + "TransportHeartbeatInterval": "00:00:05", + "TransportFailureThreshold": "00:00:15", + "CentralFetchBaseUrl": "http://10.220.153.210:8085" + }, + "ManagementService": { + "CommandTimeout": "00:00:30", + "LongRunningCommandTimeout": "00:05:00" + }, + "HealthMonitoring": { + "ReportInterval": "00:00:30", + "OfflineTimeout": "00:01:00" + }, + "InboundApi": { + "DefaultMethodTimeout": "00:00:30", + "ApiKeyStore": { + "Provider": "SqlServer" + }, + "StandbyForwardPeerEndpoints": [ + "http://10.220.153.205:8085" + ] + }, + "Notification": { + "SmtpServer": "REPLACE_WITH_SMTP_HOST", + "SmtpPort": 25, + "AuthMode": "None", + "FromAddress": "scada-notifications@zimmer.com" + }, + "NotificationOutbox": { + "DispatchInterval": "00:00:10", + "DispatchBatchSize": 100, + "StuckAgeThreshold": "00:10:00", + "TerminalRetention": "365.00:00:00", + "PurgeInterval": "1.00:00:00", + "DeliveredKpiWindow": "00:01:00" + }, + "Logging": { + "MinimumLevel": "Information" + } + }, + "Secrets": { + "Store": { + "Provider": "SqlServer" + } + }, + "Serilog": { + "WriteTo": [ + {}, + { + "Args": { + "path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log" + } + } + ] + } +} \ No newline at end of file diff --git a/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp03 b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp03 new file mode 100644 index 0000000..4282ff6 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp03 @@ -0,0 +1,89 @@ +{ + "ScadaBridge": { + "Node": { + "Role": "Site", + "NodeName": "node-a", + "NodeHostname": "10.201.75.86", + "SiteId": "__Q8_SITE_ID__", + "RemotingPort": 8082, + "GrpcPort": 8083, + "MetricsPort": 8084 + }, + "Cluster": { + "SeedNodes": [ + "akka.tcp://scadabridge@10.201.75.86:8082", + "akka.tcp://scadabridge@10.201.75.87:8082" + ], + "SplitBrainResolverStrategy": "leader-auto-down", + "StableAfter": "00:00:15", + "HeartbeatInterval": "00:00:02", + "FailureDetectionThreshold": "00:00:10", + "MinNrOfMembers": 1, + "BootstrapGuard": { + "Enabled": true + } + }, + "_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.", + "Database": { + "SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db" + }, + "DataConnection": { + "ReconnectInterval": "00:00:05", + "TagResolutionRetryInterval": "00:00:10", + "WriteTimeout": "00:00:30" + }, + "StoreAndForward": { + "SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db" + }, + "Communication": { + "CentralGrpcEndpoints": [ + "http://10.220.153.205:8083", + "http://10.220.153.210:8083" + ], + "_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-.", + "DeploymentTimeout": "00:02:00", + "LifecycleTimeout": "00:00:30", + "QueryTimeout": "00:00:30", + "TransportHeartbeatInterval": "00:00:05", + "TransportFailureThreshold": "00:00:15" + }, + "HealthMonitoring": { + "ReportInterval": "00:00:30", + "OfflineTimeout": "00:01:00" + }, + "SiteEventLog": { + "DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db", + "RetentionDays": 30, + "MaxStorageMb": 1024, + "PurgeScheduleCron": "0 2 * * *" + }, + "Notification": {}, + "Logging": { + "MinimumLevel": "Information" + } + }, + "Serilog": { + "WriteTo": [ + {}, + { + "Args": { + "path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log" + } + } + ] + }, + "LocalDb": { + "Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db", + "Replication": { + "PeerAddress": "http://10.201.75.87:8083", + "MaxBatchSize": 16, + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" + } + }, + "AuditLog": { + "SiteWriter": { + "DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db" + } + } +} \ No newline at end of file diff --git a/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp04 b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp04 new file mode 100644 index 0000000..08e7d8f --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/appsettings.Site.json.prod-parvp04 @@ -0,0 +1,88 @@ +{ + "ScadaBridge": { + "Node": { + "Role": "Site", + "NodeName": "node-b", + "NodeHostname": "10.201.75.87", + "SiteId": "__Q8_SITE_ID__", + "RemotingPort": 8082, + "GrpcPort": 8083, + "MetricsPort": 8084 + }, + "Cluster": { + "SeedNodes": [ + "akka.tcp://scadabridge@10.201.75.87:8082", + "akka.tcp://scadabridge@10.201.75.86:8082" + ], + "SplitBrainResolverStrategy": "leader-auto-down", + "StableAfter": "00:00:15", + "HeartbeatInterval": "00:00:02", + "FailureDetectionThreshold": "00:00:10", + "MinNrOfMembers": 1, + "BootstrapGuard": { + "Enabled": true + } + }, + "_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.", + "Database": { + "SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db" + }, + "DataConnection": { + "ReconnectInterval": "00:00:05", + "TagResolutionRetryInterval": "00:00:10", + "WriteTimeout": "00:00:30" + }, + "StoreAndForward": { + "SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db" + }, + "Communication": { + "CentralGrpcEndpoints": [ + "http://10.220.153.205:8083", + "http://10.220.153.210:8083" + ], + "_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-.", + "DeploymentTimeout": "00:02:00", + "LifecycleTimeout": "00:00:30", + "QueryTimeout": "00:00:30", + "TransportHeartbeatInterval": "00:00:05", + "TransportFailureThreshold": "00:00:15" + }, + "HealthMonitoring": { + "ReportInterval": "00:00:30", + "OfflineTimeout": "00:01:00" + }, + "SiteEventLog": { + "DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db", + "RetentionDays": 30, + "MaxStorageMb": 1024, + "PurgeScheduleCron": "0 2 * * *" + }, + "Notification": {}, + "Logging": { + "MinimumLevel": "Information" + } + }, + "Serilog": { + "WriteTo": [ + {}, + { + "Args": { + "path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log" + } + } + ] + }, + "LocalDb": { + "Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db", + "Replication": { + "MaxBatchSize": 16, + "MaxOplogRows": 250000, + "MaxOplogAge": "2.00:00:00" + } + }, + "AuditLog": { + "SiteWriter": { + "DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db" + } + } +} \ No newline at end of file diff --git a/docs/deploy/2026-11-19-prod-stage/sb/install-sb.ps1 b/docs/deploy/2026-11-19-prod-stage/sb/install-sb.ps1 new file mode 100644 index 0000000..668b3e2 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/install-sb.ps1 @@ -0,0 +1,110 @@ +# PROD ScadaBridge first install of ONE node (plan Phase 5 = Central on vp03/vp04, Phase 6 = Site on par-vp03/04). +# Run ON the box, only after the owner's go for that phase. Never starts the service. +# -Mode dry : verify the zip, pick this box's config, check placeholders + values, expand into .\dry\, print the +# service definition (env names only). Creates nothing outside this stage folder. +# -Mode real : expand into E:\ApiInstall\ScadaBridge\{app,cli}, put this node's appsettings..json in place, +# create \{data,logs}, the KEK (or -KekFrom; ONE scadabridge.kek for all four nodes), and the +# ScadaBridge- service (LocalSystem, auto, env REG_MULTI_SZ, recovery 5s/5s/10s, failureflag 1). +# -ValuesDir holds one .txt per '@file' line of sb-.env; never printed; the caller deletes it afterwards. +param( + [Parameter(Mandatory)][ValidateSet('Central','Site')][string]$Role, + [Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode, + [string]$ValuesDir, + [string]$KekFrom +) +$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue' +$stage = $PSScriptRoot +$zip = Join-Path $stage 'scadabridge-2e0ce81e.zip' +$zipSha = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23' +$root = 'E:\ApiInstall\ScadaBridge' +$kekDir = 'C:\ProgramData\ZB\kek' +$kek = "$kekDir\scadabridge.kek" +$svc = "ScadaBridge-$Role" +function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m } +function Fail($m) { throw "ABORT: $m" } + +$boxMap = @{ 'WONDER-APP-VP03' = @('vp03','Central'); 'WONDER-APP-VP04' = @('vp04','Central') + 'WONDER-PAR-VP03' = @('parvp03','Site'); 'WONDER-PAR-VP04' = @('parvp04','Site') } +$b = $boxMap[$env:COMPUTERNAME.ToUpperInvariant()] +if (-not $b) { Fail "unknown box $env:COMPUTERNAME" } +if ($b[1] -ne $Role) { Fail "$env:COMPUTERNAME is a $($b[1]) box in the PROD topology, not $Role" } +Say "install-sb role=$Role mode=$Mode box=$env:COMPUTERNAME" + +# 1. artifact +$h = (Get-FileHash $zip -Algorithm SHA256).Hash +if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" } +Say 'zip sha256 ok' +# 2. this node's config +$cfg = Join-Path $stage "appsettings.$Role.json.prod-$($b[0])" +$cfgText = [IO.File]::ReadAllText($cfg) +if ($cfgText -match '__Q\d+_[A-Z_]+__') { Fail "unresolved placeholder $($Matches[0]) in $(Split-Path $cfg -Leaf)" } +if ($cfgText -match '_(DEV|QA)"') { Fail 'a _DEV/_QA name in a PROD config' } +$null = $cfgText | ConvertFrom-Json +Say "config $(Split-Path $cfg -Leaf) ok" +# 3. env +$lines = Get-Content (Join-Path $stage "sb-$($Role.ToLowerInvariant()).env") +$envList = New-Object System.Collections.Generic.List[string] +foreach ($l in $lines) { + if ($l -match '^\s*(#|$)') { continue } + $k, $v = $l -split '=', 2 + if ($v -match '__Q\d+_') { Fail "unresolved placeholder in env $k" } + if ($v -eq '@file') { + if (-not $ValuesDir) { Fail "$k needs -ValuesDir" } + $f = Join-Path $ValuesDir "$k.txt" + if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" } + $v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n") + if (-not $v) { Fail "empty value file for $k" } + if ($k -like '*JwtSigningKey' -and $v.Length -lt 32) { Fail 'JwtSigningKey shorter than 32 characters' } + } + $envList.Add("$k=$v") +} +Say "env entries: $($envList.Count) ($((($envList | ForEach-Object { ($_ -split '=')[0] }) -join ', ')))" + +# 4. layout +$base = if ($Mode -eq 'dry') { Join-Path $stage 'dry' } else { $root } +$app = Join-Path $base 'app'; $cli = Join-Path $base 'cli' +if ($Mode -eq 'real') { + if (Test-Path $app) { Fail "$app already exists - first-install script only" } + if (Get-Service $svc -ErrorAction SilentlyContinue) { Fail "service $svc already exists" } +} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force } +New-Item -ItemType Directory -Force $base | Out-Null +Expand-Archive -LiteralPath $zip -DestinationPath $base +Copy-Item $cfg (Join-Path $app "appsettings.$Role.json") -Force +$roleDir = Join-Path $base $Role.ToLowerInvariant() +New-Item -ItemType Directory -Force (Join-Path $roleDir 'data'), (Join-Path $roleDir 'logs') | Out-Null +$exe = Join-Path $app 'ZB.MOM.WW.ScadaBridge.Host.exe' +Say "expanded to ${base}: app $(@(Get-ChildItem $app -Recurse -File).Count) files, cli $(@(Get-ChildItem $cli -Recurse -File).Count) files; Host $((Get-Item $exe).VersionInfo.ProductVersion)" + +# 5. KEK +if ($Mode -eq 'real') { + New-Item -ItemType Directory -Force $kekDir | Out-Null + & icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null + if (Test-Path $kek) { Say "KEK exists, kept" } + elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say 'KEK copied from -KekFrom' } + else { + if ($b[0] -ne 'vp03') { Fail 'only vp03 generates scadabridge.kek; every other node takes the same file via -KekFrom' } + $r = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($r); [IO.File]::WriteAllBytes($kek, $r); Say 'KEK generated' + } + if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' } + Say "KEK sha256 $((Get-FileHash $kek).Hash.Substring(0,8)) (must match on all four nodes)" +} + +# 6. service, never started +$disp = "ScadaBridge $Role Node" +if ($Mode -eq 'dry') { + Say "[DRYRUN] sc.exe create $svc binPath= `"$exe`" start= auto obj= LocalSystem DisplayName= `"$disp`"" + Say "[DRYRUN] HKLM\...\Services\$svc Environment = REG_MULTI_SZ, $($envList.Count) entries (values hidden)" + Say "[DRYRUN] sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000; sc.exe failureflag $svc 1" +} else { + & sc.exe create $svc binPath= "`"$(Join-Path $root 'app\ZB.MOM.WW.ScadaBridge.Host.exe')`"" start= auto obj= LocalSystem DisplayName= $disp | Out-Null + if ($LASTEXITCODE -ne 0) { Fail "sc create exited $LASTEXITCODE" } + Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc" -Name Environment -Type MultiString -Value $envList.ToArray() + & sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null + & sc.exe failureflag $svc 1 | Out-Null + $n = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc").Environment).Count + if ($n -ne $envList.Count) { Fail "env read-back $n entries, expected $($envList.Count)" } + Say "service $svc created STOPPED, env $n entries" + if ($Role -eq 'Central') { Say 'Next: ConfigDb (sql\), role rows, zbsecrets schema, seed nam-ldap-svc-password ONCE, then start vp03, then vp04.' } + else { Say 'Next: SB-GRPC-PSK- in central store + Site entity; start BOTH site nodes together (plan Phase 6).' } +} +Say "DONE mode=$Mode" diff --git a/docs/deploy/2026-11-19-prod-stage/sb/sb-central.env b/docs/deploy/2026-11-19-prod-stage/sb/sb-central.env new file mode 100644 index 0000000..15d9de1 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/sb-central.env @@ -0,0 +1,15 @@ +# ScadaBridge-Central service env (REG_MULTI_SZ Environment) for PROD vp03 + vp04. QA key set minus MachineDataDb and +# the LDAP password (§3A.3) and minus SitePsks (central reads SB-GRPC-PSK- from its store, Phase 6). +# @file = \.txt at install time. JwtSigningKey and ApiKeyPepper MUST be the same on both centrals. +SCADABRIDGE_CONFIG=Central +ASPNETCORE_URLS=http://+:8085 +ScadaBridge__Database__ConfigurationDb=@file +ScadaBridge__Security__JwtSigningKey=@file +ScadaBridge__InboundApi__ApiKeyPepper=@file +ScadaBridge__InboundApi__ApiKeyStore__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\inbound-api-keys.sqlite +Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\secrets.db +Secrets__MasterKey__Source=File +Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek +Secrets__SqlServer__ConnectionString=@file +ScadaBridge__Telemetry__Exporter=Otlp +ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317 diff --git a/docs/deploy/2026-11-19-prod-stage/sb/sb-site.env b/docs/deploy/2026-11-19-prod-stage/sb/sb-site.env new file mode 100644 index 0000000..d64fa9f --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sb/sb-site.env @@ -0,0 +1,11 @@ +# ScadaBridge-Site service env for PROD par-vp03 + par-vp04 (QA key set). Both values MUST be identical on both site +# nodes; GrpcPsk also goes into central's store as SB-GRPC-PSK-. Never reuse the PSK as the LocalDb key. +# The KEK file must be the SAME scadabridge.kek as the centrals (deployed secrets are central-encrypted). +SCADABRIDGE_CONFIG=Site +ScadaBridge__Communication__GrpcPsk=@file +LocalDb__Replication__ApiKey=@file +Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\site\data\secrets.db +Secrets__MasterKey__Source=File +Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek +ScadaBridge__Telemetry__Exporter=Otlp +ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317 diff --git a/docs/deploy/2026-11-19-prod-stage/sql/sb-role-mappings-prod.sql b/docs/deploy/2026-11-19-prod-stage/sql/sb-role-mappings-prod.sql new file mode 100644 index 0000000..1437ad6 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sql/sb-role-mappings-prod.sql @@ -0,0 +1,20 @@ +-- PROD ScadaBridge LDAP group -> role rows (plan §3A.3, D-P6). Run in the PROD ScadaBridge DB after the schema script +-- (sqlcmd -I -b). Idempotent; refuses if a group is already mapped to a different role. Only _PROD names (§3A.2). +-- Later changes go through the CLI (audited). Ids 1-5 are the migration's SCADA-* seed rows; they stay untouched. +SET QUOTED_IDENTIFIER ON; +SET XACT_ABORT ON; +BEGIN TRAN; +DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL); +INSERT @t VALUES + (N'UG_WW_APP_ADMIN_PROD', N'Administrator'), + (N'UG_WW_APP_IDE_PROD', N'Designer'), + (N'UG_WW_APP_IDE_RO_PROD', N'Viewer'), + (N'UG_WW_BTMGR_RO_PROD', N'Viewer'), + (N'UG_WW_HIST_ADMIN_PROD', N'Deployer'); -- interim until UG_WW_APP_DEPLOY_PROD exists (D-P6) +IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role) + THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1; +INSERT dbo.LdapGroupMappings (LdapGroupName, Role) +SELECT t.LdapGroupName, t.Role FROM @t t +WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName); +COMMIT; +SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id; diff --git a/docs/deploy/2026-11-19-prod-stage/sql/scadabridge-schema-2e0ce81e.sql b/docs/deploy/2026-11-19-prod-stage/sql/scadabridge-schema-2e0ce81e.sql new file mode 100644 index 0000000..910fdd8 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sql/scadabridge-schema-2e0ce81e.sql @@ -0,0 +1,2113 @@ +IF OBJECT_ID(N'[__EFMigrationsHistory]') IS NULL +BEGIN + CREATE TABLE [__EFMigrationsHistory] ( + [MigrationId] nvarchar(150) NOT NULL, + [ProductVersion] nvarchar(32) NOT NULL, + CONSTRAINT [PK___EFMigrationsHistory] PRIMARY KEY ([MigrationId]) + ); +END; +GO + +BEGIN TRANSACTION; +CREATE TABLE [ApiKeys] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [KeyValue] nvarchar(500) NOT NULL, + [IsEnabled] bit NOT NULL, + CONSTRAINT [PK_ApiKeys] PRIMARY KEY ([Id]) +); + +CREATE TABLE [ApiMethods] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [Script] nvarchar(max) NOT NULL, + [ApprovedApiKeyIds] nvarchar(4000) NULL, + [ParameterDefinitions] nvarchar(4000) NULL, + [ReturnDefinition] nvarchar(4000) NULL, + [TimeoutSeconds] int NOT NULL, + CONSTRAINT [PK_ApiMethods] PRIMARY KEY ([Id]) +); + +CREATE TABLE [AuditLogEntries] ( + [Id] int NOT NULL IDENTITY, + [User] nvarchar(200) NOT NULL, + [Action] nvarchar(100) NOT NULL, + [EntityType] nvarchar(200) NOT NULL, + [EntityId] nvarchar(200) NOT NULL, + [EntityName] nvarchar(200) NOT NULL, + [AfterStateJson] nvarchar(max) NULL, + [Timestamp] datetimeoffset NOT NULL, + CONSTRAINT [PK_AuditLogEntries] PRIMARY KEY ([Id]) +); + +CREATE TABLE [DatabaseConnectionDefinitions] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [ConnectionString] nvarchar(4000) NOT NULL, + [MaxRetries] int NOT NULL, + [RetryDelay] time NOT NULL, + CONSTRAINT [PK_DatabaseConnectionDefinitions] PRIMARY KEY ([Id]) +); + +CREATE TABLE [DataConnections] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [Protocol] nvarchar(50) NOT NULL, + [Configuration] nvarchar(4000) NULL, + CONSTRAINT [PK_DataConnections] PRIMARY KEY ([Id]) +); + +CREATE TABLE [DataProtectionKeys] ( + [Id] int NOT NULL IDENTITY, + [FriendlyName] nvarchar(max) NULL, + [Xml] nvarchar(max) NULL, + CONSTRAINT [PK_DataProtectionKeys] PRIMARY KEY ([Id]) +); + +CREATE TABLE [ExternalSystemDefinitions] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [EndpointUrl] nvarchar(2000) NOT NULL, + [AuthType] nvarchar(50) NOT NULL, + [AuthConfiguration] nvarchar(4000) NULL, + [MaxRetries] int NOT NULL, + [RetryDelay] time NOT NULL, + CONSTRAINT [PK_ExternalSystemDefinitions] PRIMARY KEY ([Id]) +); + +CREATE TABLE [LdapGroupMappings] ( + [Id] int NOT NULL IDENTITY, + [LdapGroupName] nvarchar(500) NOT NULL, + [Role] nvarchar(100) NOT NULL, + CONSTRAINT [PK_LdapGroupMappings] PRIMARY KEY ([Id]) +); + +CREATE TABLE [NotificationLists] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + CONSTRAINT [PK_NotificationLists] PRIMARY KEY ([Id]) +); + +CREATE TABLE [SharedScripts] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [Code] nvarchar(max) NOT NULL, + [ParameterDefinitions] nvarchar(4000) NULL, + [ReturnDefinition] nvarchar(4000) NULL, + CONSTRAINT [PK_SharedScripts] PRIMARY KEY ([Id]) +); + +CREATE TABLE [Sites] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [SiteIdentifier] nvarchar(100) NOT NULL, + [Description] nvarchar(2000) NULL, + CONSTRAINT [PK_Sites] PRIMARY KEY ([Id]) +); + +CREATE TABLE [SmtpConfigurations] ( + [Id] int NOT NULL IDENTITY, + [Host] nvarchar(500) NOT NULL, + [Port] int NOT NULL, + [AuthType] nvarchar(50) NOT NULL, + [Credentials] nvarchar(4000) NULL, + [TlsMode] nvarchar(50) NULL, + [FromAddress] nvarchar(500) NOT NULL, + [ConnectionTimeoutSeconds] int NOT NULL, + [MaxConcurrentConnections] int NOT NULL, + [MaxRetries] int NOT NULL, + [RetryDelay] time NOT NULL, + CONSTRAINT [PK_SmtpConfigurations] PRIMARY KEY ([Id]) +); + +CREATE TABLE [SystemArtifactDeploymentRecords] ( + [Id] int NOT NULL IDENTITY, + [ArtifactType] nvarchar(100) NOT NULL, + [DeployedBy] nvarchar(200) NOT NULL, + [DeployedAt] datetimeoffset NOT NULL, + [PerSiteStatus] nvarchar(4000) NULL, + CONSTRAINT [PK_SystemArtifactDeploymentRecords] PRIMARY KEY ([Id]) +); + +CREATE TABLE [Templates] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [Description] nvarchar(2000) NULL, + [ParentTemplateId] int NULL, + CONSTRAINT [PK_Templates] PRIMARY KEY ([Id]), + CONSTRAINT [FK_Templates_Templates_ParentTemplateId] FOREIGN KEY ([ParentTemplateId]) REFERENCES [Templates] ([Id]) ON DELETE NO ACTION +); + +CREATE TABLE [ExternalSystemMethods] ( + [Id] int NOT NULL IDENTITY, + [ExternalSystemDefinitionId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [HttpMethod] nvarchar(10) NOT NULL, + [Path] nvarchar(2000) NOT NULL, + [ParameterDefinitions] nvarchar(4000) NULL, + [ReturnDefinition] nvarchar(4000) NULL, + CONSTRAINT [PK_ExternalSystemMethods] PRIMARY KEY ([Id]), + CONSTRAINT [FK_ExternalSystemMethods_ExternalSystemDefinitions_ExternalSystemDefinitionId] FOREIGN KEY ([ExternalSystemDefinitionId]) REFERENCES [ExternalSystemDefinitions] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [NotificationRecipients] ( + [Id] int NOT NULL IDENTITY, + [NotificationListId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [EmailAddress] nvarchar(500) NOT NULL, + CONSTRAINT [PK_NotificationRecipients] PRIMARY KEY ([Id]), + CONSTRAINT [FK_NotificationRecipients_NotificationLists_NotificationListId] FOREIGN KEY ([NotificationListId]) REFERENCES [NotificationLists] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [Areas] ( + [Id] int NOT NULL IDENTITY, + [SiteId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [ParentAreaId] int NULL, + CONSTRAINT [PK_Areas] PRIMARY KEY ([Id]), + CONSTRAINT [FK_Areas_Areas_ParentAreaId] FOREIGN KEY ([ParentAreaId]) REFERENCES [Areas] ([Id]) ON DELETE NO ACTION, + CONSTRAINT [FK_Areas_Sites_SiteId] FOREIGN KEY ([SiteId]) REFERENCES [Sites] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [SiteDataConnectionAssignments] ( + [Id] int NOT NULL IDENTITY, + [SiteId] int NOT NULL, + [DataConnectionId] int NOT NULL, + CONSTRAINT [PK_SiteDataConnectionAssignments] PRIMARY KEY ([Id]), + CONSTRAINT [FK_SiteDataConnectionAssignments_DataConnections_DataConnectionId] FOREIGN KEY ([DataConnectionId]) REFERENCES [DataConnections] ([Id]) ON DELETE CASCADE, + CONSTRAINT [FK_SiteDataConnectionAssignments_Sites_SiteId] FOREIGN KEY ([SiteId]) REFERENCES [Sites] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [SiteScopeRules] ( + [Id] int NOT NULL IDENTITY, + [LdapGroupMappingId] int NOT NULL, + [SiteId] int NOT NULL, + CONSTRAINT [PK_SiteScopeRules] PRIMARY KEY ([Id]), + CONSTRAINT [FK_SiteScopeRules_LdapGroupMappings_LdapGroupMappingId] FOREIGN KEY ([LdapGroupMappingId]) REFERENCES [LdapGroupMappings] ([Id]) ON DELETE CASCADE, + CONSTRAINT [FK_SiteScopeRules_Sites_SiteId] FOREIGN KEY ([SiteId]) REFERENCES [Sites] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [TemplateAlarms] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [Description] nvarchar(2000) NULL, + [PriorityLevel] int NOT NULL, + [IsLocked] bit NOT NULL, + [TriggerType] nvarchar(50) NOT NULL, + [TriggerConfiguration] nvarchar(4000) NULL, + [OnTriggerScriptId] int NULL, + CONSTRAINT [PK_TemplateAlarms] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateAlarms_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [TemplateAttributes] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [Value] nvarchar(4000) NULL, + [DataType] nvarchar(50) NOT NULL, + [IsLocked] bit NOT NULL, + [Description] nvarchar(2000) NULL, + [DataSourceReference] nvarchar(500) NULL, + CONSTRAINT [PK_TemplateAttributes] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateAttributes_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [TemplateCompositions] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [ComposedTemplateId] int NOT NULL, + [InstanceName] nvarchar(200) NOT NULL, + CONSTRAINT [PK_TemplateCompositions] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateCompositions_Templates_ComposedTemplateId] FOREIGN KEY ([ComposedTemplateId]) REFERENCES [Templates] ([Id]) ON DELETE NO ACTION, + CONSTRAINT [FK_TemplateCompositions_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [TemplateScripts] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [IsLocked] bit NOT NULL, + [Code] nvarchar(max) NOT NULL, + [TriggerType] nvarchar(50) NULL, + [TriggerConfiguration] nvarchar(4000) NULL, + [ParameterDefinitions] nvarchar(4000) NULL, + [ReturnDefinition] nvarchar(4000) NULL, + [MinTimeBetweenRuns] time NULL, + CONSTRAINT [PK_TemplateScripts] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateScripts_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [Instances] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [SiteId] int NOT NULL, + [AreaId] int NULL, + [UniqueName] nvarchar(200) NOT NULL, + [State] nvarchar(50) NOT NULL, + CONSTRAINT [PK_Instances] PRIMARY KEY ([Id]), + CONSTRAINT [FK_Instances_Areas_AreaId] FOREIGN KEY ([AreaId]) REFERENCES [Areas] ([Id]) ON DELETE SET NULL, + CONSTRAINT [FK_Instances_Sites_SiteId] FOREIGN KEY ([SiteId]) REFERENCES [Sites] ([Id]) ON DELETE NO ACTION, + CONSTRAINT [FK_Instances_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE NO ACTION +); + +CREATE TABLE [DeployedConfigSnapshots] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [DeploymentId] nvarchar(100) NOT NULL, + [RevisionHash] nvarchar(100) NOT NULL, + [ConfigurationJson] nvarchar(max) NOT NULL, + [DeployedAt] datetimeoffset NOT NULL, + CONSTRAINT [PK_DeployedConfigSnapshots] PRIMARY KEY ([Id]), + CONSTRAINT [FK_DeployedConfigSnapshots_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [DeploymentRecords] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [Status] nvarchar(50) NOT NULL, + [DeploymentId] nvarchar(100) NOT NULL, + [RevisionHash] nvarchar(100) NULL, + [DeployedBy] nvarchar(200) NOT NULL, + [DeployedAt] datetimeoffset NOT NULL, + [CompletedAt] datetimeoffset NULL, + [ErrorMessage] nvarchar(max) NULL, + [RowVersion] rowversion NOT NULL, + CONSTRAINT [PK_DeploymentRecords] PRIMARY KEY ([Id]), + CONSTRAINT [FK_DeploymentRecords_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE NO ACTION +); + +CREATE TABLE [InstanceAttributeOverrides] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [AttributeName] nvarchar(200) NOT NULL, + [OverrideValue] nvarchar(4000) NULL, + CONSTRAINT [PK_InstanceAttributeOverrides] PRIMARY KEY ([Id]), + CONSTRAINT [FK_InstanceAttributeOverrides_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [InstanceConnectionBindings] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [AttributeName] nvarchar(200) NOT NULL, + [DataConnectionId] int NOT NULL, + CONSTRAINT [PK_InstanceConnectionBindings] PRIMARY KEY ([Id]), + CONSTRAINT [FK_InstanceConnectionBindings_DataConnections_DataConnectionId] FOREIGN KEY ([DataConnectionId]) REFERENCES [DataConnections] ([Id]) ON DELETE NO ACTION, + CONSTRAINT [FK_InstanceConnectionBindings_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +IF EXISTS (SELECT * FROM [sys].[identity_columns] WHERE [name] IN (N'Id', N'LdapGroupName', N'Role') AND [object_id] = OBJECT_ID(N'[LdapGroupMappings]')) + SET IDENTITY_INSERT [LdapGroupMappings] ON; +INSERT INTO [LdapGroupMappings] ([Id], [LdapGroupName], [Role]) +VALUES (1, N'SCADA-Admins', N'Admin'); +IF EXISTS (SELECT * FROM [sys].[identity_columns] WHERE [name] IN (N'Id', N'LdapGroupName', N'Role') AND [object_id] = OBJECT_ID(N'[LdapGroupMappings]')) + SET IDENTITY_INSERT [LdapGroupMappings] OFF; + +CREATE UNIQUE INDEX [IX_ApiKeys_KeyValue] ON [ApiKeys] ([KeyValue]); + +CREATE UNIQUE INDEX [IX_ApiKeys_Name] ON [ApiKeys] ([Name]); + +CREATE UNIQUE INDEX [IX_ApiMethods_Name] ON [ApiMethods] ([Name]); + +CREATE INDEX [IX_Areas_ParentAreaId] ON [Areas] ([ParentAreaId]); + +CREATE UNIQUE INDEX [IX_Areas_SiteId_ParentAreaId_Name] ON [Areas] ([SiteId], [ParentAreaId], [Name]) WHERE [ParentAreaId] IS NOT NULL; + +CREATE INDEX [IX_AuditLogEntries_Action] ON [AuditLogEntries] ([Action]); + +CREATE INDEX [IX_AuditLogEntries_EntityId] ON [AuditLogEntries] ([EntityId]); + +CREATE INDEX [IX_AuditLogEntries_EntityType] ON [AuditLogEntries] ([EntityType]); + +CREATE INDEX [IX_AuditLogEntries_Timestamp] ON [AuditLogEntries] ([Timestamp]); + +CREATE INDEX [IX_AuditLogEntries_User] ON [AuditLogEntries] ([User]); + +CREATE UNIQUE INDEX [IX_DatabaseConnectionDefinitions_Name] ON [DatabaseConnectionDefinitions] ([Name]); + +CREATE UNIQUE INDEX [IX_DataConnections_Name] ON [DataConnections] ([Name]); + +CREATE INDEX [IX_DeployedConfigSnapshots_DeploymentId] ON [DeployedConfigSnapshots] ([DeploymentId]); + +CREATE UNIQUE INDEX [IX_DeployedConfigSnapshots_InstanceId] ON [DeployedConfigSnapshots] ([InstanceId]); + +CREATE INDEX [IX_DeploymentRecords_DeployedAt] ON [DeploymentRecords] ([DeployedAt]); + +CREATE UNIQUE INDEX [IX_DeploymentRecords_DeploymentId] ON [DeploymentRecords] ([DeploymentId]); + +CREATE INDEX [IX_DeploymentRecords_InstanceId] ON [DeploymentRecords] ([InstanceId]); + +CREATE UNIQUE INDEX [IX_ExternalSystemDefinitions_Name] ON [ExternalSystemDefinitions] ([Name]); + +CREATE UNIQUE INDEX [IX_ExternalSystemMethods_ExternalSystemDefinitionId_Name] ON [ExternalSystemMethods] ([ExternalSystemDefinitionId], [Name]); + +CREATE UNIQUE INDEX [IX_InstanceAttributeOverrides_InstanceId_AttributeName] ON [InstanceAttributeOverrides] ([InstanceId], [AttributeName]); + +CREATE INDEX [IX_InstanceConnectionBindings_DataConnectionId] ON [InstanceConnectionBindings] ([DataConnectionId]); + +CREATE UNIQUE INDEX [IX_InstanceConnectionBindings_InstanceId_AttributeName] ON [InstanceConnectionBindings] ([InstanceId], [AttributeName]); + +CREATE INDEX [IX_Instances_AreaId] ON [Instances] ([AreaId]); + +CREATE UNIQUE INDEX [IX_Instances_SiteId_UniqueName] ON [Instances] ([SiteId], [UniqueName]); + +CREATE INDEX [IX_Instances_TemplateId] ON [Instances] ([TemplateId]); + +CREATE UNIQUE INDEX [IX_LdapGroupMappings_LdapGroupName] ON [LdapGroupMappings] ([LdapGroupName]); + +CREATE UNIQUE INDEX [IX_NotificationLists_Name] ON [NotificationLists] ([Name]); + +CREATE INDEX [IX_NotificationRecipients_NotificationListId] ON [NotificationRecipients] ([NotificationListId]); + +CREATE UNIQUE INDEX [IX_SharedScripts_Name] ON [SharedScripts] ([Name]); + +CREATE INDEX [IX_SiteDataConnectionAssignments_DataConnectionId] ON [SiteDataConnectionAssignments] ([DataConnectionId]); + +CREATE UNIQUE INDEX [IX_SiteDataConnectionAssignments_SiteId_DataConnectionId] ON [SiteDataConnectionAssignments] ([SiteId], [DataConnectionId]); + +CREATE UNIQUE INDEX [IX_Sites_Name] ON [Sites] ([Name]); + +CREATE UNIQUE INDEX [IX_Sites_SiteIdentifier] ON [Sites] ([SiteIdentifier]); + +CREATE UNIQUE INDEX [IX_SiteScopeRules_LdapGroupMappingId_SiteId] ON [SiteScopeRules] ([LdapGroupMappingId], [SiteId]); + +CREATE INDEX [IX_SiteScopeRules_SiteId] ON [SiteScopeRules] ([SiteId]); + +CREATE INDEX [IX_SystemArtifactDeploymentRecords_DeployedAt] ON [SystemArtifactDeploymentRecords] ([DeployedAt]); + +CREATE UNIQUE INDEX [IX_TemplateAlarms_TemplateId_Name] ON [TemplateAlarms] ([TemplateId], [Name]); + +CREATE UNIQUE INDEX [IX_TemplateAttributes_TemplateId_Name] ON [TemplateAttributes] ([TemplateId], [Name]); + +CREATE INDEX [IX_TemplateCompositions_ComposedTemplateId] ON [TemplateCompositions] ([ComposedTemplateId]); + +CREATE UNIQUE INDEX [IX_TemplateCompositions_TemplateId_InstanceName] ON [TemplateCompositions] ([TemplateId], [InstanceName]); + +CREATE UNIQUE INDEX [IX_Templates_Name] ON [Templates] ([Name]); + +CREATE INDEX [IX_Templates_ParentTemplateId] ON [Templates] ([ParentTemplateId]); + +CREATE UNIQUE INDEX [IX_TemplateScripts_TemplateId_Name] ON [TemplateScripts] ([TemplateId], [Name]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260317065749_InitialSchema', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Sites] ADD [NodeAAddress] nvarchar(500) NULL; + +ALTER TABLE [Sites] ADD [NodeBAddress] nvarchar(500) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260318025613_AddSiteNodeAddresses', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Sites] ADD [GrpcNodeAAddress] nvarchar(max) NULL; + +ALTER TABLE [Sites] ADD [GrpcNodeBAddress] nvarchar(max) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260321164227_AddGrpcNodeAddresses', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DROP INDEX [IX_DataConnections_Name] ON [DataConnections]; + +ALTER TABLE [DataConnections] ADD [SiteId] int NULL; + +EXEC(N' + -- Phase A: Assign the first site to each existing DataConnection + UPDATE dc + SET dc.SiteId = a.SiteId + FROM DataConnections dc + INNER JOIN ( + SELECT DataConnectionId, MIN(SiteId) AS SiteId + FROM SiteDataConnectionAssignments + GROUP BY DataConnectionId + ) a ON dc.Id = a.DataConnectionId + WHERE dc.SiteId IS NULL; + + -- Phase B: For connections assigned to additional sites, create copies + -- and update InstanceConnectionBindings to point to the new copy + DECLARE @AssignSiteId INT, @AssignConnId INT, @NewConnId INT; + DECLARE @OrigName NVARCHAR(200), @OrigProtocol NVARCHAR(50), @OrigConfig NVARCHAR(4000); + + DECLARE assignment_cursor CURSOR FOR + SELECT a.SiteId, a.DataConnectionId + FROM SiteDataConnectionAssignments a + INNER JOIN DataConnections dc ON a.DataConnectionId = dc.Id + WHERE dc.SiteId <> a.SiteId; + + OPEN assignment_cursor; + FETCH NEXT FROM assignment_cursor INTO @AssignSiteId, @AssignConnId; + + WHILE @@FETCH_STATUS = 0 + BEGIN + SELECT @OrigName = Name, @OrigProtocol = Protocol, @OrigConfig = Configuration + FROM DataConnections WHERE Id = @AssignConnId; + + INSERT INTO DataConnections (SiteId, Name, Protocol, Configuration) + VALUES (@AssignSiteId, @OrigName, @OrigProtocol, @OrigConfig); + + SET @NewConnId = SCOPE_IDENTITY(); + + -- Update bindings for instances on this site to point to the new connection + UPDATE icb + SET icb.DataConnectionId = @NewConnId + FROM InstanceConnectionBindings icb + INNER JOIN Instances i ON icb.InstanceId = i.Id + WHERE icb.DataConnectionId = @AssignConnId + AND i.SiteId = @AssignSiteId; + + FETCH NEXT FROM assignment_cursor INTO @AssignSiteId, @AssignConnId; + END + + CLOSE assignment_cursor; + DEALLOCATE assignment_cursor; + + -- Phase C: Handle any DataConnections not assigned to any site + -- (assign to the first site as a fallback) + UPDATE dc + SET dc.SiteId = (SELECT TOP 1 Id FROM Sites ORDER BY Id) + FROM DataConnections dc + WHERE dc.SiteId IS NULL; + '); + +DECLARE @var nvarchar(max); +SELECT @var = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[DataConnections]') AND [c].[name] = N'SiteId'); +IF @var IS NOT NULL EXEC(N'ALTER TABLE [DataConnections] DROP CONSTRAINT ' + @var + ';'); +UPDATE [DataConnections] SET [SiteId] = 0 WHERE [SiteId] IS NULL; +ALTER TABLE [DataConnections] ALTER COLUMN [SiteId] int NOT NULL; +ALTER TABLE [DataConnections] ADD DEFAULT 0 FOR [SiteId]; + +CREATE UNIQUE INDEX [IX_DataConnections_SiteId_Name] ON [DataConnections] ([SiteId], [Name]); + +ALTER TABLE [DataConnections] ADD CONSTRAINT [FK_DataConnections_Sites_SiteId] FOREIGN KEY ([SiteId]) REFERENCES [Sites] ([Id]) ON DELETE NO ACTION; + +DROP TABLE [SiteDataConnectionAssignments]; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260321210402_AddSiteIdToDataConnections', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +EXEC sp_rename N'[DataConnections].[Configuration]', N'PrimaryConfiguration', 'COLUMN'; + +ALTER TABLE [DataConnections] ADD [BackupConfiguration] nvarchar(4000) NULL; + +ALTER TABLE [DataConnections] ADD [FailoverRetryCount] int NOT NULL DEFAULT 3; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260511144149_AddPrimaryBackupDataConnections', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Templates] ADD [FolderId] int NULL; + +CREATE TABLE [TemplateFolders] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [ParentFolderId] int NULL, + [SortOrder] int NOT NULL, + CONSTRAINT [PK_TemplateFolders] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateFolders_TemplateFolders_ParentFolderId] FOREIGN KEY ([ParentFolderId]) REFERENCES [TemplateFolders] ([Id]) ON DELETE NO ACTION +); + +CREATE INDEX [IX_Templates_FolderId] ON [Templates] ([FolderId]); + +CREATE UNIQUE INDEX [IX_TemplateFolders_ParentFolderId_Name] ON [TemplateFolders] ([ParentFolderId], [Name]) WHERE [ParentFolderId] IS NOT NULL; + +ALTER TABLE [Templates] ADD CONSTRAINT [FK_Templates_TemplateFolders_FolderId] FOREIGN KEY ([FolderId]) REFERENCES [TemplateFolders] ([Id]) ON DELETE NO ACTION; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260511144239_AddTemplateFolders', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateScripts] ADD [IsInherited] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [TemplateScripts] ADD [LockedInDerived] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [Templates] ADD [IsDerived] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [Templates] ADD [OwnerCompositionId] int NULL; + +ALTER TABLE [TemplateAttributes] ADD [IsInherited] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [TemplateAttributes] ADD [LockedInDerived] bit NOT NULL DEFAULT CAST(0 AS bit); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260512121446_AddDerivedTemplateFields', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +SET NOCOUNT ON; + +DECLARE @collisions NVARCHAR(MAX) = ( + SELECT STRING_AGG(owner.Name + N'.' + c.InstanceName, N', ') + FROM TemplateCompositions c + INNER JOIN Templates base_t ON base_t.Id = c.ComposedTemplateId + INNER JOIN Templates owner ON owner.Id = c.TemplateId + INNER JOIN Templates existing ON existing.Name = owner.Name + N'.' + c.InstanceName + WHERE base_t.IsDerived = 0 +); +IF @collisions IS NOT NULL +BEGIN + DECLARE @msg NVARCHAR(MAX) = + N'MigrateCompositionsToDerived: cannot create derived templates — these names already exist: ' + + @collisions + + N'. Rename the conflicting templates and retry the migration.'; + THROW 50000, @msg, 1; +END + +DECLARE @CompId INT, @BaseId INT, @OwnerName NVARCHAR(200), @SlotName NVARCHAR(200); +DECLARE @NewId INT, @NewName NVARCHAR(200); + +DECLARE map_cursor CURSOR FAST_FORWARD FOR + SELECT c.Id, c.ComposedTemplateId, owner.Name, c.InstanceName + FROM TemplateCompositions c + INNER JOIN Templates base_t ON base_t.Id = c.ComposedTemplateId + INNER JOIN Templates owner ON owner.Id = c.TemplateId + WHERE base_t.IsDerived = 0; + +OPEN map_cursor; +FETCH NEXT FROM map_cursor INTO @CompId, @BaseId, @OwnerName, @SlotName; + +WHILE @@FETCH_STATUS = 0 +BEGIN + SET @NewName = @OwnerName + N'.' + @SlotName; + + INSERT INTO Templates (Name, Description, ParentTemplateId, FolderId, IsDerived, OwnerCompositionId) + SELECT @NewName, b.Description, b.Id, NULL, 1, @CompId + FROM Templates b + WHERE b.Id = @BaseId; + + SET @NewId = SCOPE_IDENTITY(); + + INSERT INTO TemplateAttributes + (TemplateId, Name, Value, DataType, IsLocked, Description, DataSourceReference, IsInherited, LockedInDerived) + SELECT @NewId, a.Name, a.Value, a.DataType, a.IsLocked, a.Description, a.DataSourceReference, 1, 0 + FROM TemplateAttributes a + WHERE a.TemplateId = @BaseId; + + INSERT INTO TemplateScripts + (TemplateId, Name, Code, IsLocked, TriggerType, TriggerConfiguration, ParameterDefinitions, ReturnDefinition, MinTimeBetweenRuns, IsInherited, LockedInDerived) + SELECT @NewId, s.Name, s.Code, s.IsLocked, s.TriggerType, s.TriggerConfiguration, s.ParameterDefinitions, s.ReturnDefinition, s.MinTimeBetweenRuns, 1, 0 + FROM TemplateScripts s + WHERE s.TemplateId = @BaseId; + + UPDATE TemplateCompositions + SET ComposedTemplateId = @NewId + WHERE Id = @CompId; + + FETCH NEXT FROM map_cursor INTO @CompId, @BaseId, @OwnerName, @SlotName; +END + +CLOSE map_cursor; +DEALLOCATE map_cursor; + + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260512122746_MigrateCompositionsToDerived', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF OBJECT_ID('dbo.fn_LegacyTypeToJsonSchemaType', 'FN') IS NOT NULL + DROP FUNCTION dbo.fn_LegacyTypeToJsonSchemaType; + + +EXEC(N' +CREATE FUNCTION dbo.fn_LegacyTypeToJsonSchemaType(@legacy NVARCHAR(50)) +RETURNS NVARCHAR(50) +AS +BEGIN + RETURN + CASE LOWER(ISNULL(@legacy, ''string'')) + WHEN ''boolean'' THEN ''boolean'' + WHEN ''bool'' THEN ''boolean'' + WHEN ''integer'' THEN ''integer'' + WHEN ''int'' THEN ''integer'' + WHEN ''int32'' THEN ''integer'' + WHEN ''int64'' THEN ''integer'' + WHEN ''float'' THEN ''number'' + WHEN ''double'' THEN ''number'' + WHEN ''decimal'' THEN ''number'' + WHEN ''number'' THEN ''number'' + WHEN ''string'' THEN ''string'' + WHEN ''datetime'' THEN ''string'' + WHEN ''object'' THEN ''object'' + WHEN ''list'' THEN ''array'' + WHEN ''array'' THEN ''array'' + ELSE ''string'' + END; +END; +'); + + +IF OBJECT_ID('dbo.fn_LegacyParametersToJsonSchema', 'FN') IS NOT NULL + DROP FUNCTION dbo.fn_LegacyParametersToJsonSchema; + + +EXEC(N' +CREATE FUNCTION dbo.fn_LegacyParametersToJsonSchema(@legacy NVARCHAR(MAX)) +RETURNS NVARCHAR(MAX) +AS +BEGIN + IF @legacy IS NULL OR LTRIM(@legacy) = '''' RETURN NULL; + IF LEFT(LTRIM(@legacy), 1) <> ''['' RETURN @legacy; -- already schema-shaped + + DECLARE @props NVARCHAR(MAX) = ( + SELECT STRING_AGG( + CONCAT( + ''"'', + STRING_ESCAPE(JSON_VALUE(p.value, ''$.name''), ''json''), + ''":'', + CASE + WHEN LOWER(ISNULL(JSON_VALUE(p.value, ''$.type''), ''string'')) IN (''list'', ''array'') + THEN CONCAT( + ''{"type":"array","items":{"type":"'', + dbo.fn_LegacyTypeToJsonSchemaType(JSON_VALUE(p.value, ''$.itemType'')), + ''"}}'') + ELSE CONCAT( + ''{"type":"'', + dbo.fn_LegacyTypeToJsonSchemaType(JSON_VALUE(p.value, ''$.type'')), + ''"}'') + END), + '','') + WITHIN GROUP (ORDER BY p.[key]) + FROM OPENJSON(@legacy) p + WHERE JSON_VALUE(p.value, ''$.name'') IS NOT NULL + AND JSON_VALUE(p.value, ''$.name'') <> '''' + ); + + DECLARE @required NVARCHAR(MAX) = ( + SELECT STRING_AGG( + CONCAT(''"'', STRING_ESCAPE(JSON_VALUE(p.value, ''$.name''), ''json''), ''"''), + '','') + WITHIN GROUP (ORDER BY p.[key]) + FROM OPENJSON(@legacy) p + WHERE JSON_VALUE(p.value, ''$.name'') IS NOT NULL + AND JSON_VALUE(p.value, ''$.name'') <> '''' + AND LOWER(ISNULL(JSON_VALUE(p.value, ''$.required''), ''true'')) <> ''false'' + ); + + RETURN + ''{"type":"object","properties":{'' + ISNULL(@props, '''') + ''}'' + + CASE WHEN @required IS NULL OR @required = '''' THEN '''' + ELSE '',"required":['' + @required + '']'' + END + + ''}''; +END; +'); + + +IF OBJECT_ID('dbo.fn_LegacyReturnToJsonSchema', 'FN') IS NOT NULL + DROP FUNCTION dbo.fn_LegacyReturnToJsonSchema; + + +EXEC(N' +CREATE FUNCTION dbo.fn_LegacyReturnToJsonSchema(@legacy NVARCHAR(MAX)) +RETURNS NVARCHAR(MAX) +AS +BEGIN + IF @legacy IS NULL OR LTRIM(@legacy) = '''' RETURN NULL; + IF LEFT(LTRIM(@legacy), 1) <> ''{'' RETURN @legacy; + + DECLARE @legacyType NVARCHAR(50) = JSON_VALUE(@legacy, ''$.type''); + IF @legacyType IS NULL RETURN @legacy; + + -- Already JSON Schema (lowercase types, no itemType legacy sentinel): leave it. + IF @legacyType IN (''boolean'',''integer'',''number'',''string'',''object'',''array'') + AND JSON_VALUE(@legacy, ''$.itemType'') IS NULL + RETURN @legacy; + + IF LOWER(@legacyType) = ''list'' + BEGIN + DECLARE @inner NVARCHAR(50) = + dbo.fn_LegacyTypeToJsonSchemaType(JSON_VALUE(@legacy, ''$.itemType'')); + RETURN CONCAT(''{"type":"array","items":{"type":"'', @inner, ''"}}''); + END; + + RETURN CONCAT(''{"type":"'', dbo.fn_LegacyTypeToJsonSchemaType(@legacyType), ''"}''); +END; +'); + +EXEC(N' +UPDATE TemplateScripts +SET ParameterDefinitions = dbo.fn_LegacyParametersToJsonSchema(ParameterDefinitions) +WHERE ParameterDefinitions IS NOT NULL + AND LEFT(LTRIM(ParameterDefinitions), 1) = ''[''; + +UPDATE TemplateScripts +SET ReturnDefinition = dbo.fn_LegacyReturnToJsonSchema(ReturnDefinition) +WHERE ReturnDefinition IS NOT NULL + AND LEFT(LTRIM(ReturnDefinition), 1) = ''{''; + +UPDATE SharedScripts +SET ParameterDefinitions = dbo.fn_LegacyParametersToJsonSchema(ParameterDefinitions) +WHERE ParameterDefinitions IS NOT NULL + AND LEFT(LTRIM(ParameterDefinitions), 1) = ''[''; + +UPDATE SharedScripts +SET ReturnDefinition = dbo.fn_LegacyReturnToJsonSchema(ReturnDefinition) +WHERE ReturnDefinition IS NOT NULL + AND LEFT(LTRIM(ReturnDefinition), 1) = ''{''; + +UPDATE ApiMethods +SET ParameterDefinitions = dbo.fn_LegacyParametersToJsonSchema(ParameterDefinitions) +WHERE ParameterDefinitions IS NOT NULL + AND LEFT(LTRIM(ParameterDefinitions), 1) = ''[''; + +UPDATE ApiMethods +SET ReturnDefinition = dbo.fn_LegacyReturnToJsonSchema(ReturnDefinition) +WHERE ReturnDefinition IS NOT NULL + AND LEFT(LTRIM(ReturnDefinition), 1) = ''{''; +'); + + +DROP FUNCTION IF EXISTS dbo.fn_LegacyParametersToJsonSchema; +DROP FUNCTION IF EXISTS dbo.fn_LegacyReturnToJsonSchema; +DROP FUNCTION IF EXISTS dbo.fn_LegacyTypeToJsonSchemaType; + + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260512211204_MigrateParametersToJsonSchema', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [InstanceAlarmOverrides] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [AlarmCanonicalName] nvarchar(400) NOT NULL, + [TriggerConfigurationOverride] nvarchar(4000) NULL, + [PriorityLevelOverride] int NULL, + CONSTRAINT [PK_InstanceAlarmOverrides] PRIMARY KEY ([Id]), + CONSTRAINT [FK_InstanceAlarmOverrides_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +CREATE UNIQUE INDEX [IX_InstanceAlarmOverrides_InstanceId_AlarmCanonicalName] ON [InstanceAlarmOverrides] ([InstanceId], [AlarmCanonicalName]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260513055537_AddInstanceAlarmOverrides', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateAlarms] ADD [IsInherited] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [TemplateAlarms] ADD [LockedInDerived] bit NOT NULL DEFAULT CAST(0 AS bit); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260517000628_AddDerivedAlarmFields', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DECLARE @var1 nvarchar(max); +SELECT @var1 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[SmtpConfigurations]') AND [c].[name] = N'Credentials'); +IF @var1 IS NOT NULL EXEC(N'ALTER TABLE [SmtpConfigurations] DROP CONSTRAINT ' + @var1 + ';'); +ALTER TABLE [SmtpConfigurations] ALTER COLUMN [Credentials] nvarchar(max) NULL; + +DECLARE @var2 nvarchar(max); +SELECT @var2 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[ExternalSystemDefinitions]') AND [c].[name] = N'AuthConfiguration'); +IF @var2 IS NOT NULL EXEC(N'ALTER TABLE [ExternalSystemDefinitions] DROP CONSTRAINT ' + @var2 + ';'); +ALTER TABLE [ExternalSystemDefinitions] ALTER COLUMN [AuthConfiguration] nvarchar(max) NULL; + +DECLARE @var3 nvarchar(max); +SELECT @var3 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[DatabaseConnectionDefinitions]') AND [c].[name] = N'ConnectionString'); +IF @var3 IS NOT NULL EXEC(N'ALTER TABLE [DatabaseConnectionDefinitions] DROP CONSTRAINT ' + @var3 + ';'); +ALTER TABLE [DatabaseConnectionDefinitions] ALTER COLUMN [ConnectionString] nvarchar(max) NOT NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260517010521_EncryptSecretColumns', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DECLARE @var4 nvarchar(max); +SELECT @var4 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[Sites]') AND [c].[name] = N'GrpcNodeBAddress'); +IF @var4 IS NOT NULL EXEC(N'ALTER TABLE [Sites] DROP CONSTRAINT ' + @var4 + ';'); +ALTER TABLE [Sites] ALTER COLUMN [GrpcNodeBAddress] nvarchar(500) NULL; + +DECLARE @var5 nvarchar(max); +SELECT @var5 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[Sites]') AND [c].[name] = N'GrpcNodeAAddress'); +IF @var5 IS NOT NULL EXEC(N'ALTER TABLE [Sites] DROP CONSTRAINT ' + @var5 + ';'); +ALTER TABLE [Sites] ALTER COLUMN [GrpcNodeAAddress] nvarchar(500) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260517020720_BoundGrpcNodeAddressLength', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DELETE FROM ApiKeys; + +DROP INDEX [IX_ApiKeys_KeyValue] ON [ApiKeys]; + +DECLARE @var6 nvarchar(max); +SELECT @var6 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[ApiKeys]') AND [c].[name] = N'KeyValue'); +IF @var6 IS NOT NULL EXEC(N'ALTER TABLE [ApiKeys] DROP CONSTRAINT ' + @var6 + ';'); +ALTER TABLE [ApiKeys] DROP COLUMN [KeyValue]; + +ALTER TABLE [ApiKeys] ADD [KeyHash] nvarchar(256) NOT NULL DEFAULT N''; + +CREATE UNIQUE INDEX [IX_ApiKeys_KeyHash] ON [ApiKeys] ([KeyHash]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260517073000_HashApiKeyValue', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DROP INDEX [IX_Templates_Name] ON [Templates]; + + + UPDATE t + SET t.Name = c.InstanceName + FROM Templates t + INNER JOIN TemplateCompositions c ON c.Id = t.OwnerCompositionId + WHERE t.IsDerived = 1; + +CREATE UNIQUE INDEX [IX_Templates_Name] ON [Templates] ([Name]) WHERE [IsDerived] = 0; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260518214444_ContainedDerivedTemplateNames', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [NotificationLists] ADD [Type] nvarchar(32) NOT NULL DEFAULT N'Email'; + +CREATE TABLE [Notifications] ( + [NotificationId] nvarchar(64) NOT NULL, + [Type] nvarchar(32) NOT NULL, + [ListName] nvarchar(200) NOT NULL, + [Subject] nvarchar(1000) NOT NULL, + [Body] nvarchar(max) NOT NULL, + [TypeData] nvarchar(max) NULL, + [Status] nvarchar(32) NOT NULL, + [RetryCount] int NOT NULL, + [LastError] nvarchar(4000) NULL, + [ResolvedTargets] nvarchar(max) NULL, + [SourceSiteId] nvarchar(100) NOT NULL, + [SourceInstanceId] nvarchar(200) NULL, + [SourceScript] nvarchar(200) NULL, + [SiteEnqueuedAt] datetimeoffset NOT NULL, + [CreatedAt] datetimeoffset NOT NULL, + [LastAttemptAt] datetimeoffset NULL, + [NextAttemptAt] datetimeoffset NULL, + [DeliveredAt] datetimeoffset NULL, + CONSTRAINT [PK_Notifications] PRIMARY KEY ([NotificationId]) +); + +CREATE INDEX [IX_Notifications_SourceSiteId_CreatedAt] ON [Notifications] ([SourceSiteId], [CreatedAt]); + +CREATE INDEX [IX_Notifications_Status_NextAttemptAt] ON [Notifications] ([Status], [NextAttemptAt]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260519050659_AddNotificationsTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +CREATE PARTITION FUNCTION pf_AuditLog_Month (datetime2(7)) +AS RANGE RIGHT FOR VALUES ( + '2026-01-01T00:00:00', '2026-02-01T00:00:00', '2026-03-01T00:00:00', '2026-04-01T00:00:00', + '2026-05-01T00:00:00', '2026-06-01T00:00:00', '2026-07-01T00:00:00', '2026-08-01T00:00:00', + '2026-09-01T00:00:00', '2026-10-01T00:00:00', '2026-11-01T00:00:00', '2026-12-01T00:00:00', + '2027-01-01T00:00:00', '2027-02-01T00:00:00', '2027-03-01T00:00:00', '2027-04-01T00:00:00', + '2027-05-01T00:00:00', '2027-06-01T00:00:00', '2027-07-01T00:00:00', '2027-08-01T00:00:00', + '2027-09-01T00:00:00', '2027-10-01T00:00:00', '2027-11-01T00:00:00', '2027-12-01T00:00:00' +); + + +CREATE PARTITION SCHEME ps_AuditLog_Month +AS PARTITION pf_AuditLog_Month ALL TO ([PRIMARY]); + + +CREATE TABLE dbo.AuditLog ( + EventId uniqueidentifier NOT NULL, + OccurredAtUtc datetime2(7) NOT NULL, + IngestedAtUtc datetime2(7) NULL, + Channel varchar(32) NOT NULL, + Kind varchar(32) NOT NULL, + CorrelationId uniqueidentifier NULL, + SourceSiteId varchar(64) NULL, + SourceInstanceId varchar(128) NULL, + SourceScript varchar(128) NULL, + Actor varchar(128) NULL, + Target varchar(256) NULL, + Status varchar(32) NOT NULL, + HttpStatus int NULL, + DurationMs int NULL, + ErrorMessage nvarchar(1024) NULL, + ErrorDetail nvarchar(max) NULL, + RequestSummary nvarchar(max) NULL, + ResponseSummary nvarchar(max) NULL, + PayloadTruncated bit NOT NULL, + Extra nvarchar(max) NULL, + ForwardState varchar(32) NULL, + CONSTRAINT PK_AuditLog PRIMARY KEY CLUSTERED (EventId, OccurredAtUtc) + ON ps_AuditLog_Month(OccurredAtUtc) +) ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred +ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId +ON dbo.AuditLog (CorrelationId) +WHERE CorrelationId IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred +ON dbo.AuditLog (Channel ASC, Status ASC, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred +ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) +WHERE Target IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId +ON dbo.AuditLog (EventId) +ON [PRIMARY]; + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_writer'; +GRANT INSERT ON dbo.AuditLog TO scadabridge_audit_writer; +GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_writer; +DENY UPDATE ON dbo.AuditLog TO scadabridge_audit_writer; +DENY DELETE ON dbo.AuditLog TO scadabridge_audit_writer; + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_purger; +GRANT ALTER ON SCHEMA::dbo TO scadabridge_audit_purger; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260520142214_AddAuditLogTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [SiteCalls] ( + [TrackedOperationId] varchar(36) NOT NULL, + [Channel] varchar(32) NOT NULL, + [Target] varchar(256) NOT NULL, + [SourceSite] varchar(64) NOT NULL, + [Status] varchar(32) NOT NULL, + [RetryCount] int NOT NULL, + [LastError] nvarchar(1024) NULL, + [HttpStatus] int NULL, + [CreatedAtUtc] datetime2 NOT NULL, + [UpdatedAtUtc] datetime2 NOT NULL, + [TerminalAtUtc] datetime2 NULL, + [IngestedAtUtc] datetime2 NOT NULL, + CONSTRAINT [PK_SiteCalls] PRIMARY KEY ([TrackedOperationId]) +); + +CREATE INDEX [IX_SiteCalls_Source_Created] ON [SiteCalls] ([SourceSite], [CreatedAtUtc] DESC); + +CREATE INDEX [IX_SiteCalls_Status_Updated] ON [SiteCalls] ([Status], [UpdatedAtUtc] DESC); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260520180431_AddSiteCallsTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [AuditLog] ADD [ExecutionId] uniqueidentifier NULL; + +EXEC(N' +CREATE NONCLUSTERED INDEX IX_AuditLog_Execution +ON dbo.AuditLog (ExecutionId) +WHERE ExecutionId IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc);'); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260521184044_AddAuditLogExecutionId', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Notifications] ADD [OriginExecutionId] uniqueidentifier NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260521193048_AddNotificationOriginExecutionId', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [AuditLog] ADD [ParentExecutionId] uniqueidentifier NULL; + +EXEC(N' +CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution +ON dbo.AuditLog (ParentExecutionId) +WHERE ParentExecutionId IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc);'); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260521210254_AddAuditLogParentExecutionId', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Notifications] ADD [OriginParentExecutionId] uniqueidentifier NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260521220924_AddNotificationOriginParentExecutionId', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [AuditLog] ADD [SourceNode] varchar(64) NULL; + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred +ON dbo.AuditLog (SourceNode, OccurredAtUtc) +ON ps_AuditLog_Month(OccurredAtUtc); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260523201754_AddAuditLogSourceNode', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Notifications] ADD [SourceNode] varchar(64) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260523201950_AddNotificationSourceNode', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [SiteCalls] ADD [SourceNode] varchar(64) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260523202131_AddSiteCallSourceNode', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [AuditLogEntries] ADD [BundleImportId] uniqueidentifier NULL; + +CREATE INDEX [IX_AuditLogEntries_BundleImportId] ON [AuditLogEntries] ([BundleImportId]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260524075016_AddBundleImportIdToAuditLog', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [InstanceConnectionBindings] ADD [DataSourceReferenceOverride] nvarchar(512) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260528155520_AddInstanceConnectionBindingOverride', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [InstanceNativeAlarmSourceOverrides] ( + [Id] int NOT NULL IDENTITY, + [InstanceId] int NOT NULL, + [SourceCanonicalName] nvarchar(400) NOT NULL, + [ConnectionNameOverride] nvarchar(200) NULL, + [SourceReferenceOverride] nvarchar(1000) NULL, + [ConditionFilterOverride] nvarchar(1000) NULL, + CONSTRAINT [PK_InstanceNativeAlarmSourceOverrides] PRIMARY KEY ([Id]), + CONSTRAINT [FK_InstanceNativeAlarmSourceOverrides_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +CREATE TABLE [TemplateNativeAlarmSources] ( + [Id] int NOT NULL IDENTITY, + [TemplateId] int NOT NULL, + [Name] nvarchar(200) NOT NULL, + [Description] nvarchar(2000) NULL, + [ConnectionName] nvarchar(200) NOT NULL, + [SourceReference] nvarchar(1000) NOT NULL, + [ConditionFilter] nvarchar(1000) NULL, + [IsLocked] bit NOT NULL, + [IsInherited] bit NOT NULL, + [LockedInDerived] bit NOT NULL, + CONSTRAINT [PK_TemplateNativeAlarmSources] PRIMARY KEY ([Id]), + CONSTRAINT [FK_TemplateNativeAlarmSources_Templates_TemplateId] FOREIGN KEY ([TemplateId]) REFERENCES [Templates] ([Id]) ON DELETE CASCADE +); + +CREATE UNIQUE INDEX [IX_InstanceNativeAlarmSourceOverrides_InstanceId_SourceCanonicalName] ON [InstanceNativeAlarmSourceOverrides] ([InstanceId], [SourceCanonicalName]); + +CREATE UNIQUE INDEX [IX_TemplateNativeAlarmSources_TemplateId_Name] ON [TemplateNativeAlarmSources] ([TemplateId], [Name]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260529195700_AddNativeAlarmSources', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DROP TABLE [ApiKeys]; + +DECLARE @var7 nvarchar(max); +SELECT @var7 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[ApiMethods]') AND [c].[name] = N'ApprovedApiKeyIds'); +IF @var7 IS NOT NULL EXEC(N'ALTER TABLE [ApiMethods] DROP CONSTRAINT ' + @var7 + ';'); +ALTER TABLE [ApiMethods] DROP COLUMN [ApprovedApiKeyIds]; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260602092753_RetireInboundApiKeyStore', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +UPDATE [LdapGroupMappings] SET [Role] = N'Administrator' +WHERE [Id] = 1; +SELECT @@ROWCOUNT; + + +UPDATE [LdapGroupMappings] SET [Role] = N'Designer' +WHERE [Id] = 2; +SELECT @@ROWCOUNT; + + +UPDATE [LdapGroupMappings] SET [Role] = N'Deployer' +WHERE [Id] = 3; +SELECT @@ROWCOUNT; + + +UPDATE [LdapGroupMappings] SET [Role] = N'Deployer' +WHERE [Id] = 4; +SELECT @@ROWCOUNT; + + +UPDATE [LdapGroupMappings] SET [Role] = N'Administrator' WHERE [Role] IN (N'Admin', N'Audit'); + +UPDATE [LdapGroupMappings] SET [Role] = N'Designer' WHERE [Role] = N'Design'; + +UPDATE [LdapGroupMappings] SET [Role] = N'Deployer' WHERE [Role] = N'Deployment'; + +UPDATE [LdapGroupMappings] SET [Role] = N'Viewer' WHERE [Role] = N'AuditReadOnly'; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260602113822_CanonicalizeRoles', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +CREATE TABLE dbo.AuditLog_v2 ( + EventId uniqueidentifier NOT NULL, + OccurredAtUtc datetime2(7) NOT NULL, + Actor nvarchar(256) NULL, + Action varchar(64) NOT NULL, + Outcome varchar(16) NOT NULL, + Category varchar(32) NOT NULL, + Target nvarchar(256) NULL, + SourceNode varchar(64) NULL, + CorrelationId uniqueidentifier NULL, + DetailsJson nvarchar(max) NULL, + Kind AS JSON_VALUE(DetailsJson,'$.kind') PERSISTED, + Status AS JSON_VALUE(DetailsJson,'$.status') PERSISTED, + SourceSiteId AS JSON_VALUE(DetailsJson,'$.sourceSiteId') PERSISTED, + ExecutionId AS CAST(JSON_VALUE(DetailsJson,'$.executionId') AS uniqueidentifier) PERSISTED, + ParentExecutionId AS CAST(JSON_VALUE(DetailsJson,'$.parentExecutionId') AS uniqueidentifier) PERSISTED, + -- IngestedAtUtc is NOT persisted: the datetimeoffset cast / SWITCHOFFSET is + -- non-deterministic and SQL Server rejects a PERSISTED non-deterministic + -- computed column. It is not indexed, so non-persistence costs nothing. + IngestedAtUtc AS CAST(SWITCHOFFSET(CAST(JSON_VALUE(DetailsJson,'$.ingestedAtUtc') AS datetimeoffset), 0) AS datetime2(7)), + CONSTRAINT PK_AuditLog_v2 PRIMARY KEY CLUSTERED (EventId, OccurredAtUtc) + ON ps_AuditLog_Month(OccurredAtUtc) +) ON ps_AuditLog_Month(OccurredAtUtc); + + +INSERT INTO dbo.AuditLog_v2 + (EventId, OccurredAtUtc, Actor, Action, Outcome, Category, Target, SourceNode, CorrelationId, DetailsJson) +SELECT + a.EventId, + a.OccurredAtUtc, + NULLIF(a.Actor, '') AS Actor, + a.Channel + '.' + a.Kind AS Action, + CASE + WHEN a.Kind = 'InboundAuthFailure' THEN 'Denied' + WHEN a.Status IN ('Failed','Parked','Discarded') THEN 'Failure' + ELSE 'Success' + END AS Outcome, + a.Channel AS Category, + a.Target, + a.SourceNode, + a.CorrelationId, + ( + SELECT + a.Channel AS channel, + a.Kind AS kind, + a.Status AS status, + a.ExecutionId AS executionId, + a.ParentExecutionId AS parentExecutionId, + a.SourceSiteId AS sourceSiteId, + a.SourceInstanceId AS sourceInstanceId, + a.SourceScript AS sourceScript, + a.HttpStatus AS httpStatus, + a.DurationMs AS durationMs, + a.ErrorMessage AS errorMessage, + a.ErrorDetail AS errorDetail, + a.RequestSummary AS requestSummary, + a.ResponseSummary AS responseSummary, + CAST(a.PayloadTruncated AS bit) AS payloadTruncated, + a.Extra AS extra, + CASE WHEN a.IngestedAtUtc IS NULL THEN NULL + ELSE CONVERT(varchar(33), + CAST(a.IngestedAtUtc AS datetimeoffset(7)) AT TIME ZONE 'UTC', + 126) + END AS ingestedAtUtc + FOR JSON PATH, WITHOUT_ARRAY_WRAPPER + ) AS DetailsJson +FROM dbo.AuditLog a; + + +DROP TABLE dbo.AuditLog; +EXEC sp_rename 'dbo.AuditLog_v2', 'AuditLog'; +EXEC sp_rename 'PK_AuditLog_v2', 'PK_AuditLog', 'OBJECT'; + + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred +ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId +ON dbo.AuditLog (CorrelationId) +WHERE CorrelationId IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred +ON dbo.AuditLog (Category ASC, Status ASC, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred +ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) +WHERE Target IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc); + +-- IX_AuditLog_Execution / IX_AuditLog_ParentExecution are NOT filtered: SQL +-- Server forbids a filtered-index WHERE predicate from referencing a computed +-- column (ExecutionId / ParentExecutionId are persisted computed columns). An +-- unfiltered index still backs the equality lookups GetExecutionTreeAsync uses; +-- it just also indexes the NULL rows. (The pre-C5 typed columns allowed the +-- IS NOT NULL filter; the computed-column constraint does not.) +CREATE NONCLUSTERED INDEX IX_AuditLog_Execution +ON dbo.AuditLog (ExecutionId) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution +ON dbo.AuditLog (ParentExecutionId) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred +ON dbo.AuditLog (SourceNode ASC, OccurredAtUtc ASC) +ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId +ON dbo.AuditLog (EventId) +ON [PRIMARY]; + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_writer'; +GRANT INSERT ON dbo.AuditLog TO scadabridge_audit_writer; +GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_writer; +DENY UPDATE ON dbo.AuditLog TO scadabridge_audit_writer; +DENY DELETE ON dbo.AuditLog TO scadabridge_audit_writer; + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_purger; +GRANT ALTER ON SCHEMA::dbo TO scadabridge_audit_purger; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260602174346_CollapseAuditLogToCanonical', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF NOT EXISTS (SELECT 1 FROM [LdapGroupMappings] WHERE [Id] = 5) +BEGIN + SET IDENTITY_INSERT [LdapGroupMappings] ON; + INSERT INTO [LdapGroupMappings] ([Id], [LdapGroupName], [Role]) + VALUES (5, N'SCADA-Viewers', N'Viewer'); + SET IDENTITY_INSERT [LdapGroupMappings] OFF; +END + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260615171137_ResyncLdapGroupMappingSeed', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateScripts] ADD [ExecutionTimeoutSeconds] int NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260615183757_AddTemplateScriptExecutionTimeout', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF NOT EXISTS (SELECT 1 FROM sys.columns WHERE Name='ElementDataType' AND Object_ID=Object_ID('TemplateAttributes')) + ALTER TABLE [TemplateAttributes] ADD [ElementDataType] nvarchar(50) NULL; + + +IF NOT EXISTS (SELECT 1 FROM sys.columns WHERE Name='ElementDataType' AND Object_ID=Object_ID('InstanceAttributeOverrides')) + ALTER TABLE [InstanceAttributeOverrides] ADD [ElementDataType] nvarchar(50) NULL; + +ALTER TABLE [TemplateAttributes] ALTER COLUMN [Value] nvarchar(max) NULL; + +ALTER TABLE [InstanceAttributeOverrides] ALTER COLUMN [OverrideValue] nvarchar(max) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260616192131_AddListAttributeElementType', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [KpiSample] ( + [Id] bigint NOT NULL IDENTITY, + [Source] varchar(64) NOT NULL, + [Metric] varchar(64) NOT NULL, + [Scope] varchar(16) NOT NULL, + [ScopeKey] varchar(64) NULL, + [Value] float NOT NULL, + [CapturedAtUtc] datetime2 NOT NULL, + CONSTRAINT [PK_KpiSample] PRIMARY KEY ([Id]) +); + +CREATE INDEX [IX_KpiSample_Captured] ON [KpiSample] ([CapturedAtUtc]); + +CREATE INDEX [IX_KpiSample_Series] ON [KpiSample] ([Source], [Metric], [Scope], [ScopeKey], [CapturedAtUtc]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260617234323_AddKpiSampleTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [PendingSecuredWrites] ( + [Id] bigint NOT NULL IDENTITY, + [SiteId] varchar(128) NOT NULL, + [ConnectionName] varchar(128) NOT NULL, + [TagPath] varchar(512) NOT NULL, + [ValueJson] varchar(4000) NOT NULL, + [ValueType] varchar(128) NOT NULL, + [Status] varchar(32) NOT NULL, + [OperatorUser] varchar(256) NOT NULL, + [OperatorComment] varchar(1024) NULL, + [SubmittedAtUtc] datetime2 NOT NULL, + [VerifierUser] varchar(256) NULL, + [VerifierComment] varchar(1024) NULL, + [DecidedAtUtc] datetime2 NULL, + [ExecutedAtUtc] datetime2 NULL, + [ExecutionError] varchar(2048) NULL, + CONSTRAINT [PK_PendingSecuredWrites] PRIMARY KEY ([Id]) +); + +CREATE INDEX [IX_PendingSecuredWrites_Site] ON [PendingSecuredWrites] ([SiteId]); + +CREATE INDEX [IX_PendingSecuredWrites_Status_Submitted] ON [PendingSecuredWrites] ([Status], [SubmittedAtUtc]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260618060853_AddPendingSecuredWriteTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [SharedSchemas] ( + [Id] int NOT NULL IDENTITY, + [Name] nvarchar(200) NOT NULL, + [Scope] nvarchar(200) NULL, + [SchemaJson] nvarchar(max) NOT NULL, + CONSTRAINT [PK_SharedSchemas] PRIMARY KEY ([Id]) +); + +CREATE UNIQUE INDEX [IX_SharedSchemas_Name] ON [SharedSchemas] ([Name]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260618152548_AddSharedSchema', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [NotificationRecipients] ALTER COLUMN [EmailAddress] nvarchar(500) NULL; + + +IF NOT EXISTS (SELECT 1 FROM sys.columns WHERE Name='PhoneNumber' AND Object_ID=Object_ID('NotificationRecipients')) + ALTER TABLE [NotificationRecipients] ADD [PhoneNumber] nvarchar(32) NULL; + + +IF OBJECT_ID(N'dbo.SmsConfigurations', N'U') IS NULL + CREATE TABLE [SmsConfigurations] ( + [Id] int NOT NULL IDENTITY(1, 1), + [AccountSid] nvarchar(100) NOT NULL, + [AuthToken] nvarchar(max) NULL, + [FromNumber] nvarchar(32) NOT NULL, + [MessagingServiceSid] nvarchar(100) NULL, + [ApiBaseUrl] nvarchar(500) NULL, + [ConnectionTimeoutSeconds] int NOT NULL, + [MaxRetries] int NOT NULL, + [RetryDelay] time NOT NULL, + CONSTRAINT [PK_SmsConfigurations] PRIMARY KEY ([Id]) + ); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260619135543_AddSmsNotifications', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [SmsConfigurations] ALTER COLUMN [FromNumber] nvarchar(32) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260619191520_SmsFromNumberOptional', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [PendingDeployments] ( + [Id] int NOT NULL IDENTITY, + [DeploymentId] nvarchar(100) NOT NULL, + [InstanceId] int NOT NULL, + [RevisionHash] nvarchar(100) NOT NULL, + [ConfigurationJson] nvarchar(max) NOT NULL, + [Token] nvarchar(128) NOT NULL, + [CreatedAtUtc] datetimeoffset NOT NULL, + [ExpiresAtUtc] datetimeoffset NOT NULL, + CONSTRAINT [PK_PendingDeployments] PRIMARY KEY ([Id]), + CONSTRAINT [FK_PendingDeployments_Instances_InstanceId] FOREIGN KEY ([InstanceId]) REFERENCES [Instances] ([Id]) ON DELETE CASCADE +); + +CREATE UNIQUE INDEX [IX_PendingDeployments_DeploymentId] ON [PendingDeployments] ([DeploymentId]); + +CREATE INDEX [IX_PendingDeployments_ExpiresAtUtc] ON [PendingDeployments] ([ExpiresAtUtc]); + +CREATE INDEX [IX_PendingDeployments_InstanceId] ON [PendingDeployments] ([InstanceId]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260626161432_AddPendingDeployment', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +-- The switch-out dance CREATEs a staging table; ALTER ON SCHEMA::dbo alone does not +-- confer the database-level CREATE TABLE permission (review finding S3). +GRANT CREATE TABLE TO scadabridge_audit_purger; +-- The per-channel retention override (PurgeChannelOlderThanAsync) is a bounded row +-- DELETE on the maintenance path; a segregated purger principal needs this grant. +GRANT DELETE ON dbo.AuditLog TO scadabridge_audit_purger; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260709105112_FixAuditPurgerRoleGrants', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_SiteCalls_NonTerminal] ON [SiteCalls] ([CreatedAtUtc]) INCLUDE ([SourceSite], [SourceNode], [Status]) WHERE [TerminalAtUtc] IS NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260709110614_AddSiteCallsNonTerminalIndex', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [ExternalSystemDefinitions] ADD [TimeoutSeconds] int NOT NULL DEFAULT 0; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260710075422_AddExternalSystemTimeoutSeconds', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [SmtpConfigurations] ADD [OAuth2Authority] nvarchar(500) NULL; + +ALTER TABLE [SmtpConfigurations] ADD [OAuth2Scope] nvarchar(500) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260710080636_AddSmtpOAuth2AuthorityScope', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [KpiRollupHourly] ( + [Id] bigint NOT NULL IDENTITY, + [Source] varchar(64) NOT NULL, + [Metric] varchar(64) NOT NULL, + [Scope] varchar(16) NOT NULL, + [ScopeKey] varchar(64) NULL, + [HourStartUtc] datetime2 NOT NULL, + [Value] float NOT NULL, + [MinValue] float NOT NULL, + [MaxValue] float NOT NULL, + [SampleCount] int NOT NULL, + CONSTRAINT [PK_KpiRollupHourly] PRIMARY KEY ([Id]) +); + +CREATE INDEX [IX_KpiRollupHourly_Hour] ON [KpiRollupHourly] ([HourStartUtc]); + +CREATE UNIQUE INDEX [IX_KpiRollupHourly_Series] ON [KpiRollupHourly] ([Source], [Metric], [Scope], [ScopeKey], [HourStartUtc]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260710153953_AddKpiRollupHourlyTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_SiteCalls_Terminal] ON [SiteCalls] ([TerminalAtUtc]) WHERE [TerminalAtUtc] IS NOT NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260713142234_AddSiteCallsTerminalIndex', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [SmtpConfigurations] ADD [Transport] nvarchar(50) NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260810100836_AddSmtpConfigurationTransport', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_Notifications_Delivered] ON [Notifications] ([DeliveredAt]) WHERE [Status] = 'Delivered'; + + +DROP INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog; + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc +ON dbo.AuditLog (OccurredAtUtc DESC) +INCLUDE (Status) +ON ps_AuditLog_Month(OccurredAtUtc); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260814235335_AddAuditLogAndNotificationCoveringIndexes', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF EXISTS (SELECT 1 FROM sys.indexes + WHERE name = 'UX_AuditLog_EventId' AND object_id = OBJECT_ID('dbo.AuditLog')) + DROP INDEX UX_AuditLog_EventId ON dbo.AuditLog; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260815004957_AlignAuditLogEventIdUniqueness', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE TABLE [BundleUnlockAttempt] ( + [Id] bigint NOT NULL IDENTITY, + [ClientKey] varchar(64) NOT NULL, + [AttemptedAtUtc] datetime2 NOT NULL, + CONSTRAINT [PK_BundleUnlockAttempt] PRIMARY KEY ([Id]) +); + +CREATE INDEX [IX_BundleUnlockAttempt_Client_Attempted] ON [BundleUnlockAttempt] ([ClientKey], [AttemptedAtUtc]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260815154803_AddBundleUnlockAttemptTable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +DROP INDEX [IX_DeploymentRecords_DeployedAt] ON [DeploymentRecords]; + +CREATE INDEX [IX_DeploymentRecords_DeployedAt_Id] ON [DeploymentRecords] ([DeployedAt] DESC, [Id] DESC); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260815165452_AddDeploymentRecordListSortIndex', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_SiteCalls_Created_TrackedOperationId] ON [SiteCalls] ([CreatedAtUtc] DESC, [TrackedOperationId] DESC); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260819183326_AddSiteCallsListSortIndex', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_Notifications_Created_NotificationId] ON [Notifications] ([CreatedAt] DESC, [NotificationId] DESC); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260819201210_AddNotificationsListSortIndex', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [ApiMethods] ADD [TraceAllCalls] bit NOT NULL DEFAULT CAST(0 AS bit); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260906034113_AddApiMethodTraceAllCalls', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +IF EXISTS (SELECT UniqueName FROM Instances GROUP BY UniqueName HAVING COUNT(*) > 1) + THROW 51002, 'Duplicate global instance names exist. Inventory Instances grouped by UniqueName and resolve ownership before applying this upgrade.', 1; + +DROP INDEX [IX_Instances_SiteId_UniqueName] ON [Instances]; + +DECLARE @var8 nvarchar(max); +SELECT @var8 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[DataConnections]') AND [c].[name] = N'PrimaryConfiguration'); +IF @var8 IS NOT NULL EXEC(N'ALTER TABLE [DataConnections] DROP CONSTRAINT ' + @var8 + ';'); +ALTER TABLE [DataConnections] ALTER COLUMN [PrimaryConfiguration] nvarchar(max) NULL; + +DECLARE @var9 nvarchar(max); +SELECT @var9 = QUOTENAME([d].[name]) +FROM [sys].[default_constraints] [d] +INNER JOIN [sys].[columns] [c] ON [d].[parent_column_id] = [c].[column_id] AND [d].[parent_object_id] = [c].[object_id] +WHERE ([d].[parent_object_id] = OBJECT_ID(N'[DataConnections]') AND [c].[name] = N'BackupConfiguration'); +IF @var9 IS NOT NULL EXEC(N'ALTER TABLE [DataConnections] DROP CONSTRAINT ' + @var9 + ';'); +ALTER TABLE [DataConnections] ALTER COLUMN [BackupConfiguration] nvarchar(max) NULL; + +CREATE INDEX [IX_Instances_SiteId] ON [Instances] ([SiteId]); + +CREATE UNIQUE INDEX [IX_Instances_UniqueName] ON [Instances] ([UniqueName]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260909203046_EnforceGlobalInstanceNamesAndProtectProtocolConfiguration', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateAlarms] ADD [IsMesReportable] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [InstanceAlarmOverrides] ADD [IsMesReportableOverride] bit NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260911135501_AddAlarmMesReportable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [Sites] ADD [ArtifactRevisionHash] nvarchar(100) NULL; + +ALTER TABLE [Sites] ADD [ArtifactsDeployedAt] datetimeoffset NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20260914173147_AddSiteArtifactRevision', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_AuditLogEntries_EntityType_Timestamp] ON [AuditLogEntries] ([EntityType], [Timestamp] DESC); + +CREATE INDEX [IX_AuditLogEntries_Timestamp_Id] ON [AuditLogEntries] ([Timestamp] DESC, [Id] DESC); + +CREATE INDEX [IX_AuditLogEntries_User_Timestamp] ON [AuditLogEntries] ([User], [Timestamp] DESC); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred +ON dbo.AuditLog (Kind, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + + +CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred +ON dbo.AuditLog (Actor, OccurredAtUtc DESC) +ON ps_AuditLog_Month(OccurredAtUtc); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261022133907_ReviewRemediationIndexes', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateNativeAlarmSources] ADD [IsMesReportable] bit NOT NULL DEFAULT CAST(0 AS bit); + +ALTER TABLE [InstanceNativeAlarmSourceOverrides] ADD [IsMesReportableOverride] bit NULL; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261109200211_AddNativeAlarmSourceMesReportable', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +CREATE INDEX [IX_DeploymentRecords_DeployedBy] ON [DeploymentRecords] ([DeployedBy]); + +CREATE INDEX [IX_DeploymentRecords_RevisionHash] ON [DeploymentRecords] ([RevisionHash]); + +CREATE INDEX [IX_AuditLogEntries_EntityName] ON [AuditLogEntries] ([EntityName]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261109203753_PrefixSearchIndexes', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [AuditLog] ADD [ChainSeq] bigint NULL; + +ALTER TABLE [AuditLog] ADD [RowHash] binary(32) NULL; + +CREATE TABLE [AuditLogChainHead] ( + [ChainMonth] date NOT NULL, + [LastSeq] bigint NOT NULL, + [LastHash] binary(32) NOT NULL, + [PurgedThroughSeq] bigint NULL, + [PurgedThroughHash] binary(32) NULL, + [UpdatedAtUtc] datetime2 NOT NULL, + CONSTRAINT [PK_AuditLogChainHead] PRIMARY KEY ([ChainMonth]) +); + +CREATE TABLE [AuditLogChainTombstone] ( + [ChainMonth] date NOT NULL, + [ChainSeq] bigint NOT NULL, + [RowHash] binary(32) NOT NULL, + [EventId] uniqueidentifier NOT NULL, + [PurgedAtUtc] datetime2 NOT NULL, + CONSTRAINT [PK_AuditLogChainTombstone] PRIMARY KEY ([ChainMonth], [ChainSeq]) +); + + +EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained +ON dbo.AuditLog (OccurredAtUtc, EventId) +WHERE ChainSeq IS NULL +ON ps_AuditLog_Month(OccurredAtUtc);'); + + +EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq +ON dbo.AuditLog (ChainSeq) +WHERE ChainSeq IS NOT NULL +ON ps_AuditLog_Month(OccurredAtUtc);'); + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_chainer'; +-- The chaining job reads unchained rows and writes ONLY the two link columns. +GRANT SELECT ON dbo.AuditLog TO scadabridge_audit_chainer; +GRANT UPDATE (ChainSeq, RowHash) ON dbo.AuditLog TO scadabridge_audit_chainer; +GRANT SELECT, INSERT, UPDATE ON dbo.AuditLogChainHead TO scadabridge_audit_chainer; +GRANT SELECT ON dbo.AuditLogChainTombstone TO scadabridge_audit_chainer; + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +-- The per-channel purge writes a tombstone per chained row it removes; the partition +-- switch moves the month's head to its checkpoint anchor and drops its tombstones. +GRANT SELECT, UPDATE ON dbo.AuditLogChainHead TO scadabridge_audit_purger; +GRANT SELECT, INSERT, DELETE ON dbo.AuditLogChainTombstone TO scadabridge_audit_purger; + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_writer') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_writer'; +-- The ingest principal must never be able to forge chain state. +DENY INSERT, UPDATE, DELETE ON dbo.AuditLogChainHead TO scadabridge_audit_writer; +DENY INSERT, UPDATE, DELETE ON dbo.AuditLogChainTombstone TO scadabridge_audit_writer; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261109211317_AddAuditLogHashChain', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; +ALTER TABLE [TemplateScripts] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [Templates] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [TemplateNativeAlarmSources] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [TemplateCompositions] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [TemplateAttributes] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [TemplateAlarms] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [Sites] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [SharedScripts] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [Instances] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [InstanceNativeAlarmSourceOverrides] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [InstanceConnectionBindings] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [InstanceAttributeOverrides] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [InstanceAlarmOverrides] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [ExternalSystemMethods] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [ExternalSystemDefinitions] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [DeployedConfigSnapshots] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [DataConnections] ADD [RowVersion] rowversion NOT NULL; + +ALTER TABLE [DatabaseConnectionDefinitions] ADD [RowVersion] rowversion NOT NULL; + +CREATE INDEX [IX_TemplateScripts_RowVersion] ON [TemplateScripts] ([RowVersion]); + +CREATE INDEX [IX_Templates_RowVersion] ON [Templates] ([RowVersion]); + +CREATE INDEX [IX_TemplateNativeAlarmSources_RowVersion] ON [TemplateNativeAlarmSources] ([RowVersion]); + +CREATE INDEX [IX_TemplateCompositions_RowVersion] ON [TemplateCompositions] ([RowVersion]); + +CREATE INDEX [IX_TemplateAttributes_RowVersion] ON [TemplateAttributes] ([RowVersion]); + +CREATE INDEX [IX_TemplateAlarms_RowVersion] ON [TemplateAlarms] ([RowVersion]); + +CREATE INDEX [IX_Sites_RowVersion] ON [Sites] ([RowVersion]); + +CREATE INDEX [IX_SharedScripts_RowVersion] ON [SharedScripts] ([RowVersion]); + +CREATE INDEX [IX_Instances_RowVersion] ON [Instances] ([RowVersion]); + +CREATE INDEX [IX_InstanceNativeAlarmSourceOverrides_RowVersion] ON [InstanceNativeAlarmSourceOverrides] ([RowVersion]); + +CREATE INDEX [IX_InstanceConnectionBindings_RowVersion] ON [InstanceConnectionBindings] ([RowVersion]); + +CREATE INDEX [IX_InstanceAttributeOverrides_RowVersion] ON [InstanceAttributeOverrides] ([RowVersion]); + +CREATE INDEX [IX_InstanceAlarmOverrides_RowVersion] ON [InstanceAlarmOverrides] ([RowVersion]); + +CREATE INDEX [IX_ExternalSystemMethods_RowVersion] ON [ExternalSystemMethods] ([RowVersion]); + +CREATE INDEX [IX_ExternalSystemDefinitions_RowVersion] ON [ExternalSystemDefinitions] ([RowVersion]); + +CREATE INDEX [IX_DeployedConfigSnapshots_RowVersion] ON [DeployedConfigSnapshots] ([RowVersion]); + +CREATE INDEX [IX_DataConnections_RowVersion] ON [DataConnections] ([RowVersion]); + +CREATE INDEX [IX_DatabaseConnectionDefinitions_RowVersion] ON [DatabaseConnectionDefinitions] ([RowVersion]); + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261110032839_AddConfigGraphRowVersions', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF +(SELECT c.name FROM sys.index_columns ic + JOIN sys.columns c ON c.object_id = ic.object_id AND c.column_id = ic.column_id + WHERE ic.object_id = OBJECT_ID('dbo.AuditLog') AND ic.index_id = 1 AND ic.key_ordinal = 1) = N'EventId' +BEGIN + +DECLARE @dropSql nvarchar(max) = N''; +SELECT @dropSql = @dropSql + N'DROP INDEX ' + QUOTENAME(i.name) + N' ON dbo.AuditLog;' + CHAR(10) +FROM sys.indexes i +WHERE i.object_id = OBJECT_ID('dbo.AuditLog') AND i.type = 2 AND i.name IN ( + N'IX_AuditLog_OccurredAtUtc', N'IX_AuditLog_Site_Occurred', N'IX_AuditLog_CorrelationId', + N'IX_AuditLog_Channel_Status_Occurred', N'IX_AuditLog_Target_Occurred', N'IX_AuditLog_Execution', + N'IX_AuditLog_ParentExecution', N'IX_AuditLog_Node_Occurred', N'IX_AuditLog_Kind_Occurred', + N'IX_AuditLog_Actor_Occurred', N'IX_AuditLog_Unchained', N'IX_AuditLog_ChainSeq', + N'UX_AuditLog_EventId_Occurred'); +EXEC sp_executesql @dropSql; + ALTER TABLE dbo.AuditLog DROP CONSTRAINT PK_AuditLog; + ALTER TABLE dbo.AuditLog ADD CONSTRAINT PK_AuditLog PRIMARY KEY CLUSTERED (OccurredAtUtc, EventId) + ON ps_AuditLog_Month(OccurredAtUtc); + CREATE UNIQUE NONCLUSTERED INDEX UX_AuditLog_EventId_Occurred ON dbo.AuditLog (EventId, OccurredAtUtc) + ON ps_AuditLog_Month(OccurredAtUtc); + +CREATE NONCLUSTERED INDEX IX_AuditLog_OccurredAtUtc ON dbo.AuditLog (OccurredAtUtc DESC) INCLUDE (Status) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Site_Occurred ON dbo.AuditLog (SourceSiteId ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_CorrelationId ON dbo.AuditLog (CorrelationId) WHERE CorrelationId IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Channel_Status_Occurred ON dbo.AuditLog (Category ASC, Status ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Target_Occurred ON dbo.AuditLog (Target ASC, OccurredAtUtc DESC) WHERE Target IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Execution ON dbo.AuditLog (ExecutionId) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_ParentExecution ON dbo.AuditLog (ParentExecutionId) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Node_Occurred ON dbo.AuditLog (SourceNode, OccurredAtUtc) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Kind_Occurred ON dbo.AuditLog (Kind ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc); +CREATE NONCLUSTERED INDEX IX_AuditLog_Actor_Occurred ON dbo.AuditLog (Actor ASC, OccurredAtUtc DESC) ON ps_AuditLog_Month(OccurredAtUtc); +-- The two hash-chain indexes go through EXEC (compiled only when run): RemoveAuditLogHashChain +-- (2026-11-15) drops ChainSeq, and the idempotent script compiles this migration's batch against +-- that schema on a re-run, where a direct reference is Msg 207 (ScadaBridge#126). Same indexes. +EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_Unchained ON dbo.AuditLog (OccurredAtUtc, EventId) WHERE ChainSeq IS NULL ON ps_AuditLog_Month(OccurredAtUtc);'); +EXEC(N'CREATE NONCLUSTERED INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog (ChainSeq) WHERE ChainSeq IS NOT NULL ON ps_AuditLog_Month(OccurredAtUtc);'); +END + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261110033732_FlipAuditLogClusteredKey', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NULL + EXEC sp_executesql N'CREATE ROLE scadabridge_audit_purger'; +GRANT ALTER ANY DATASPACE TO scadabridge_audit_purger; +GRANT UPDATE (SourceNode) ON dbo.AuditLog TO scadabridge_audit_purger; + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261110120000_AuditMaintenanceGrants', N'10.0.13'); + +COMMIT; +GO + +BEGIN TRANSACTION; + +IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_ChainSeq' AND object_id = OBJECT_ID('dbo.AuditLog')) + EXEC(N'DROP INDEX IX_AuditLog_ChainSeq ON dbo.AuditLog;'); +IF EXISTS (SELECT 1 FROM sys.indexes WHERE name = 'IX_AuditLog_Unchained' AND object_id = OBJECT_ID('dbo.AuditLog')) + EXEC(N'DROP INDEX IX_AuditLog_Unchained ON dbo.AuditLog;'); + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_chainer') IS NOT NULL +BEGIN + IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL + EXEC(N'REVOKE UPDATE (ChainSeq, RowHash) ON dbo.AuditLog FROM scadabridge_audit_chainer;'); + EXEC(N'REVOKE SELECT ON dbo.AuditLog FROM scadabridge_audit_chainer;'); + + DECLARE @dropMembers nvarchar(max) = N''; + SELECT @dropMembers = @dropMembers + N'ALTER ROLE scadabridge_audit_chainer DROP MEMBER ' + QUOTENAME(m.name) + N';' + CHAR(10) + FROM sys.database_role_members rm + JOIN sys.database_principals r ON r.principal_id = rm.role_principal_id + JOIN sys.database_principals m ON m.principal_id = rm.member_principal_id + WHERE r.name = N'scadabridge_audit_chainer'; + EXEC sp_executesql @dropMembers; + + EXEC(N'DROP ROLE scadabridge_audit_chainer;'); +END + + +IF OBJECT_ID('dbo.AuditLogChainHead', 'U') IS NOT NULL + EXEC(N'DROP TABLE dbo.AuditLogChainHead;'); +IF OBJECT_ID('dbo.AuditLogChainTombstone', 'U') IS NOT NULL + EXEC(N'DROP TABLE dbo.AuditLogChainTombstone;'); + + +IF COL_LENGTH('dbo.AuditLog', 'ChainSeq') IS NOT NULL + EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN ChainSeq;'); +IF COL_LENGTH('dbo.AuditLog', 'RowHash') IS NOT NULL + EXEC(N'ALTER TABLE dbo.AuditLog DROP COLUMN RowHash;'); + + +IF DATABASE_PRINCIPAL_ID('scadabridge_audit_purger') IS NOT NULL +BEGIN + EXEC(N'REVOKE ALTER ANY DATASPACE FROM scadabridge_audit_purger;'); + EXEC(N'REVOKE UPDATE (SourceNode) ON dbo.AuditLog FROM scadabridge_audit_purger;'); +END + +INSERT INTO [__EFMigrationsHistory] ([MigrationId], [ProductVersion]) +VALUES (N'20261115221558_RemoveAuditLogHashChain', N'10.0.13'); + +COMMIT; +GO + diff --git a/docs/deploy/2026-11-19-prod-stage/sql/zbsecrets-schema-v1.sql b/docs/deploy/2026-11-19-prod-stage/sql/zbsecrets-schema-v1.sql new file mode 100644 index 0000000..ec8b273 --- /dev/null +++ b/docs/deploy/2026-11-19-prod-stage/sql/zbsecrets-schema-v1.sql @@ -0,0 +1,44 @@ +-- ScadaBridge Central shared secret store (Secrets:Store:Provider=SqlServer), schema "zbsecrets", version 1. +-- Verbatim DDL from ZB.MOM.WW.Secrets.Replicator.SqlServer SqlServerSecretsSchema.CreateSchemaDdl (unchanged since 0.2.0, +-- so identical in the 0.6.2 package ScadaBridge 2e0ce81e pins), plus the version-row MERGE its migrator writes. +-- WHY: the app creates this at first start, but nam-ldap-svc-password must be seeded BEFORE the first start (§3A.4) and the +-- headless CLI runs no migrator. Idempotent; run in the ScadaBridge DB after scadabridge-schema-2e0ce81e.sql. +SET QUOTED_IDENTIFIER ON; +IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'zbsecrets') + EXEC(N'CREATE SCHEMA [zbsecrets]'); +GO +IF OBJECT_ID(N'[zbsecrets].[schema_version]', N'U') IS NULL + CREATE TABLE [zbsecrets].[schema_version] ( + id INT NOT NULL PRIMARY KEY CHECK (id = 1), + version INT NOT NULL, + applied_utc NVARCHAR(33) NOT NULL + ); +IF OBJECT_ID(N'[zbsecrets].[secret]', N'U') IS NULL + CREATE TABLE [zbsecrets].[secret] ( + name NVARCHAR(450) NOT NULL PRIMARY KEY, + description NVARCHAR(MAX) NULL, + content_type NVARCHAR(64) NOT NULL, + ciphertext VARBINARY(MAX) NOT NULL, + nonce VARBINARY(MAX) NOT NULL, + tag VARBINARY(MAX) NOT NULL, + wrapped_dek VARBINARY(MAX) NOT NULL, + wrap_nonce VARBINARY(MAX) NOT NULL, + wrap_tag VARBINARY(MAX) NOT NULL, + kek_id NVARCHAR(256) NOT NULL, + revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0, + is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0, + deleted_utc NVARCHAR(33) NULL, + created_utc NVARCHAR(33) NOT NULL, + updated_utc NVARCHAR(33) NOT NULL, + created_by NVARCHAR(256) NULL, + updated_by NVARCHAR(256) NULL + ); +IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = N'ix_secret_updated_utc' AND object_id = OBJECT_ID(N'[zbsecrets].[secret]')) + CREATE INDEX ix_secret_updated_utc ON [zbsecrets].[secret] (updated_utc); +MERGE [zbsecrets].[schema_version] AS target +USING (SELECT 1 AS id, 1 AS version, CONVERT(nvarchar(33), FORMAT(SYSUTCDATETIME(), 'yyyy-MM-ddTHH:mm:ss.fffffff') + N'+00:00') AS applied_utc) AS source +ON target.id = source.id +WHEN NOT MATCHED THEN INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc); +GO +SELECT s.name AS [schema], t.name AS [table] FROM sys.tables t JOIN sys.schemas s ON s.schema_id = t.schema_id WHERE s.name = N'zbsecrets'; +SELECT id, version, applied_utc FROM [zbsecrets].[schema_version]; diff --git a/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md index 7b298dd..7229e2e 100644 --- a/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md +++ b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md @@ -1,6 +1,8 @@ # PROD install: ScadaBridge + MxAccessGateway (plan, 2026-11-19) -**State: PLAN ONLY. Nothing has been executed, and no prod box has been touched.** This is the first production +**State: PLAN + STAGED. Nothing has been executed on any prod box.** 2026-11-19: the install pack (builds, configs, +scripts, SQL) was copied to all four boxes over SMB only, at the owner's request; see +[`docs/deploy/2026-11-19-prod-stage/`](../deploy/2026-11-19-prod-stage/README.md). This is the first production tier for the family. Until now `deployment.md` said "no production box has been identified". **The DEPLOY HOLD still applies to PROD** (owner, 2026-11-17). DEV and QA have both run the candidate builds with NAM AD @@ -149,8 +151,9 @@ the prod integration's shape before you assume it matches QA's. that a full-history `--idempotent` script fails at batch compile. 2. Review it, then run it with `sqlcmd -I -b` (QUOTED_IDENTIFIER, or the filtered indexes fail with Msg 1934) from a box that has `sqlcmd`, using the service login. Alternatively, a DBA runs it. -3. Gate: 65 rows in `__EFMigrationsHistory`, the `zbauth` schema present (inbound API keys, Auth - `ApiKeys.SqlServer`), and the Secrets SqlServer store tables present. +3. Gate: 65 rows in `__EFMigrationsHistory`. (Correction 2026-11-19: `zbauth` and `zbsecrets` are not EF migrations; the + apps create them at first start. `zbsecrets` is needed earlier, for the §3A.4 seed, so run the staged + `sql\zbsecrets-schema-v1.sql` here; `zbauth` is left to the first central start.) 4. First backup: `COPY_ONLY, CHECKSUM, COMPRESSION` + `VERIFYONLY`. 5. Insert the `_PROD` role rows by SQL (§3A.3). Gate: the rows read back; ids 1–5 (`SCADA-*`) untouched.