deploy(prod): stage the PROD install pack on app-vp03/04 + par-vp03/04 over SMB (nothing executed)
Configs, env templates, install wrappers (dry/real, never start), SQL (65-migration schema, zbsecrets v1, _PROD role rows), preflight, seed script, patched mxgw Install script (worker name, file KEK, connstr mask). Plan: Phase 3 gate corrected. Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
# 2026-11-19: PROD install pack staged on wonder-app-vp03/vp04 + wonder-par-vp03/vp04 (SMB only)
|
||||
|
||||
**Owner request (2026-11-19):** "prep the application files, scripts, configs and copy them to the 4 servers for prod".
|
||||
Plan: [`docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md`](../../plans/2026-11-19-prod-scadabridge-mxgw-install.md).
|
||||
|
||||
**What was done:** files copied over SMB (445) as `NAM\adm_dohertj2`. **Nothing ran on any PROD box**: no WinRM, no
|
||||
service, no install, no KEK, no secret. The PROD deploy hold and every phase gate in the plan still apply. Each script
|
||||
below is run on the box only after the owner's go for its phase.
|
||||
|
||||
## Artifacts (the QA-soaked builds, byte-identical; plan D-P2)
|
||||
|
||||
| File | sha256 | Source |
|
||||
|---|---|---|
|
||||
| `scadabridge-2e0ce81e.zip` (app + cli, self-contained win-x64) | `6c706b0a…6572e23` | the zip QA vq03/vq04 run since 2026-11-19 |
|
||||
| `mxgw-e74918cc.zip` (Server + x86 Worker `MxGateway.Worker.exe`, `0.1.2+e74918cc`) | `f0999b75…5d847f2` | the zip DEV vd03 + QA run |
|
||||
| `otelcol-contrib_0.162.0_windows_amd64.zip` | `97a675ee…e496a167` | QA vq03 `E:\psrem-stage\otel-54` (scadaproj#54) |
|
||||
| Secrets CLI 0.6.6 `ZB.MOM.WW.Secrets.Cli.exe` | `36dbd512…` | = QA `E:\ApiInstall\_tools\secret-0.6.6` |
|
||||
| Secrets CLI 0.6.6 + SqlServer provider (centrals only) | `421f95e9…` | = QA `…\secret-0.6.6-sql` (QA finding 2) |
|
||||
| `nssm.exe` | `eee9c44c…` | = QA `C:\ProgramData\ZB\qa-build\tools\nssm.exe` |
|
||||
|
||||
Every staged folder carries a `SHA256SUMS.txt`. Remote sizes were checked file by file over SMB (150/150), and the two
|
||||
app zips plus the CLI were pulled back from vp04 and par-vp04 and re-hashed.
|
||||
|
||||
## Layout on each box
|
||||
|
||||
| Path | vp03 / vp04 | par-vp03 / par-vp04 |
|
||||
|---|---|---|
|
||||
| `E:\psrem-stage\prod-common\` | `nssm.exe`, `preflight-prod.ps1`, `seed-secret-prod.ps1` | same |
|
||||
| `E:\psrem-stage\mxgw-e74918cc\` | zip, `appsettings.json.prod`, `mxgw.env`, `install-mxgw.ps1`, `Install-`/`Upgrade-MxGatewayService.ps1` | same (install refuses until MXAccess exists, Q3) |
|
||||
| `E:\psrem-stage\sb-2e0ce81e\` | zip, `appsettings.Central.json.prod-<box>`, `sb-central.env`, `install-sb.ps1`, `sql\` | zip, `appsettings.Site.json.prod-<box>`, `sb-site.env`, `install-sb.ps1` |
|
||||
| `E:\psrem-stage\otel-54\` | collector zip, `config.yaml`, `Install-OtelCollector.ps1` (PROD tier added), `Test-OtelCollector.ps1` | same |
|
||||
| `E:\ApiInstall\_tools\` | `secret-0.6.6\`, `secret-0.6.6-sql\` | `secret-0.6.6\` |
|
||||
|
||||
`E:\ApiInstall` now exists only for `_tools`; there is still no `app` folder and no service, so both install scripts see a
|
||||
greenfield box.
|
||||
|
||||
## Configs (sources in this folder)
|
||||
|
||||
- **mxgw** `mxgw.env`: QA vq03's key set plus the §3A.3 LDAP block. Differences: `ASPNETCORE_ENVIRONMENT=Production` (login
|
||||
hard-stops on), `_PROD` `GroupToRole`, and the file KEK `C:\ProgramData\ZB\kek\mxgateway.kek`. `appsettings.json.prod` = the
|
||||
zip's `Server\appsettings.json` minus `Dashboard:GroupToRole` (`GwAdmin`/`GwReader`).
|
||||
- **ScadaBridge centrals**: per node `appsettings.Central.json.prod-vp03|vp04` (QA shape, IPv4 literals, self-first seeds,
|
||||
`leader-auto-down`, BootstrapGuard on, NAM LDAPS block, `RequireHttpsCookie=false`, shared SQL secret + API-key stores,
|
||||
CentralGrpcPort left at the default **8083**: QA's 8086 only existed because of co-location). `sb-central.env` is QA's key
|
||||
set minus `MachineDataDb` (gone since 3808b49f), minus the LDAP password env (§3A.3), and minus `SitePsks`
|
||||
(central reads `SB-GRPC-PSK-<siteId>` from its store, Phase 6).
|
||||
- **ScadaBridge sites**: `appsettings.Site.json.prod-parvp03|parvp04` (node-a/node-b, `PeerAddress` on node-a only, both
|
||||
centrals at `:8083`, absolute data paths). `sb-site.env` = QA's site key set.
|
||||
- **SQL (centrals, `sql\`)**: `scadabridge-schema-2e0ce81e.sql` (full, non-idempotent, 65 migrations, from `dotnet ef
|
||||
migrations script 0 20261115221558_RemoveAuditLogHashChain` at 2e0ce81e; run with `sqlcmd -I -b`),
|
||||
`zbsecrets-schema-v1.sql`, `sb-role-mappings-prod.sql` (the five `_PROD` rows, D-P6).
|
||||
|
||||
### Placeholders still open (the install scripts refuse while any remains)
|
||||
|
||||
| Placeholder / value | Where | Question |
|
||||
|---|---|---|
|
||||
| `__Q2_ALARM_EXPRESSION__` | `mxgw.env` | Q2 alarm area (`\\<GR node>\Galaxy!<area>`) |
|
||||
| `MxGateway__Galaxy__ConnectionString` (value file) | install time | Q2 prod GR SQL |
|
||||
| `__Q8_SITE_ID__` | both `appsettings.Site.json.prod-*` | Q8 SiteId |
|
||||
| `ScadaBridge__Database__ConfigurationDb`, `Secrets__SqlServer__ConnectionString` (value files) | install time | Q1 prod SQL + `scadabridge_svc` |
|
||||
| JWT key, inbound pepper (same on both centrals), mxgw pepper (shared by the PAR pair, pattern a), site PSK + LocalDb key (same on both sites) | install time, value files | generated then, stored in the Keychain |
|
||||
|
||||
When a placeholder is resolved, edit the file here, commit, and re-push that one file (its `SHA256SUMS.txt` line too).
|
||||
|
||||
## Scripts (dry first, always)
|
||||
|
||||
| Script | Does | Never does |
|
||||
|---|---|---|
|
||||
| `prod-common\preflight-prod.ps1 [-SqlHost] [-GrHost]` | read-only Phase 2 gates: disk, `LmxProxy.dll` + COM registration, NAM 636/389 + rootca + TLS, greenfield paths and ports, DNS, TCP path probes per §4 (open / refused / timeout), staged hashes | write anything |
|
||||
| `mxgw-e74918cc\install-mxgw.ps1 -Mode dry\|real [-ValuesDir] [-KekFrom]` | MXAccess gate, zip hash, placeholder + `_DEV/_QA` refusal, expand to `app`/`worker`, NAM base config, 32-byte KEK, NSSM service via the patched `Install-MxGatewayService.ps1` | start the service |
|
||||
| `sb-2e0ce81e\install-sb.ps1 -Role Central\|Site -Mode dry\|real [-ValuesDir] [-KekFrom]` | box/role check, zip hash, this node's config, env from value files, expand `app`/`cli`, `<role>\{data,logs}`, KEK (generated on vp03 only; every other node takes the same file), `sc.exe` service with recovery 5/5/10 s + failureflag | start the service |
|
||||
| `prod-common\seed-secret-prod.ps1 -Target MxAccessGw\|ScadaBridge-Central -ValueFile` | seeds `nam-ldap-svc-password` into a stopped service's store (central: once, vp03, shared SQL store), reads back, deletes the value file | print the value |
|
||||
|
||||
`Install-MxGatewayService.ps1` here is mxgw `e74918cc` plus three marked patches: `-WorkerExeName` (the wonder Worker is
|
||||
`MxGateway.Worker.exe`), File-KEK acceptance, and masking of connection strings in its env printout (the stock script
|
||||
prints `MxGateway__Galaxy__ConnectionString` in clear). `Upgrade-MxGatewayService.ps1` is the `START_PENDING`-patched copy QA
|
||||
used. Both repo fixes are still owed (mxaccessgw).
|
||||
|
||||
Checked on the Mac (pwsh 7, stubs for `nssm`/`icacls`): all eight scripts parse; `install-sb.ps1` dry runs pass for a
|
||||
central and a site and refuse a wrong role and an open `__Q8_SITE_ID__`; `install-mxgw.ps1` dry run refuses an open `__Q2__`
|
||||
and then runs `Install-MxGatewayService -DryRun` (39 env entries). That is not a Windows run; the on-box dry run is the gate.
|
||||
|
||||
## Order once the owner says go (plan phases)
|
||||
|
||||
1. Phase 2, each box: `preflight-prod.ps1` (needs the Q6 go for remote exec). Optional: `otel-54\Install-OtelCollector.ps1 -StageDir … -Tier PROD -NssmPath E:\psrem-stage\prod-common\nssm.exe`.
|
||||
2. Phase 3 (Q1 answered): `sql\scadabridge-schema-2e0ce81e.sql`, `sql\zbsecrets-schema-v1.sql`, `sql\sb-role-mappings-prod.sql`; backup.
|
||||
3. Phase 4, vp03/vp04 (par later, Q3): `install-mxgw.ps1 -Mode dry`, then `real`; `seed-secret-prod.ps1 -Target MxAccessGw`; start; §3A.5.
|
||||
4. Phase 5: `install-sb.ps1 -Role Central` on vp03 (KEK generated), vp04 (`-KekFrom` the vp03 file); `seed-secret-prod.ps1 -Target ScadaBridge-Central` on vp03; start vp03, then vp04; §3A.5.
|
||||
5. Phase 6: `install-sb.ps1 -Role Site` on both PAR boxes (`-KekFrom`, same PSK and LocalDb key); `SB-GRPC-PSK-<siteId>` into central; start both together.
|
||||
|
||||
## Cleanup (if PROD is abandoned)
|
||||
|
||||
`winrm smb rm -r <box> 'E$\psrem-stage\prod-common' 'E$\psrem-stage\mxgw-e74918cc' 'E$\psrem-stage\sb-2e0ce81e' 'E$\psrem-stage\otel-54' 'E$\ApiInstall\_tools'`
|
||||
on each box. The push also created the parent folders `E:\psrem-stage` and `E:\ApiInstall`; the earlier 2026-11-19
|
||||
SMB survey listed neither. Nothing else was created.
|
||||
@@ -0,0 +1,65 @@
|
||||
# PROD box preflight (plan Phase 2 gates + S3A LDAP gate). READ-ONLY: writes nothing, starts nothing, prints no secret.
|
||||
# Run on each of the four boxes after the owner's go for remote exec (Q6). Optional -SqlHost/-GrHost for Q1/Q2 reachability.
|
||||
param([string]$SqlHost, [string]$GrHost)
|
||||
$ErrorActionPreference = 'Continue'; $ProgressPreference = 'SilentlyContinue'
|
||||
$box = $env:COMPUTERNAME.ToUpperInvariant()
|
||||
"== preflight-prod $box $(Get-Date -Format s)"
|
||||
$ip = @{ 'WONDER-APP-VP03' = '10.220.153.205'; 'WONDER-APP-VP04' = '10.220.153.210'; 'WONDER-PAR-VP03' = '10.201.75.86'; 'WONDER-PAR-VP04' = '10.201.75.87' }
|
||||
$central = $box -like 'WONDER-APP-*'
|
||||
"role (planned): $(if ($central) { 'ScadaBridge Central + mxgw' } else { 'ScadaBridge Site + mxgw' })"
|
||||
Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" | ForEach-Object { "disk $($_.DeviceID) free $([math]::Round($_.FreeSpace/1GB,1)) GB of $([math]::Round($_.Size/1GB,1)) GB" }
|
||||
"os $((Get-CimInstance Win32_OperatingSystem).Caption) ps $($PSVersionTable.PSVersion)"
|
||||
|
||||
# MXAccess (Q3 gate for mxgw)
|
||||
$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll'
|
||||
if (Test-Path $lmx) { "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12)) (want 6000.0045.2657.1 / BD75A2FDE222)" } else { 'LmxProxy.dll ABSENT - mxgw cannot run here yet (Q3)' }
|
||||
$clsid = Get-ItemProperty 'Registry::HKEY_CLASSES_ROOT\WOW6432Node\CLSID\*\ProgID' -ErrorAction SilentlyContinue | Where-Object { $_.'(default)' -like 'LMXProxy.LMXProxyServer*' }
|
||||
"LMXProxy.LMXProxyServer COM registration: $(if ($clsid) { 'present' } else { 'absent' })"
|
||||
|
||||
# NAM LDAPS (S3A gate)
|
||||
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') { foreach ($p in 636,389) {
|
||||
$c = New-Object Net.Sockets.TcpClient; $ok = $c.ConnectAsync($h,$p).Wait(4000); $c.Dispose(); "reach $h`:$p $ok" } }
|
||||
$root = @(Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like '*zimmer-rootca*' })
|
||||
"zimmer-rootca in LocalMachine\Root: $($root.Count) $(($root | ForEach-Object { $_.Thumbprint + ' exp ' + $_.NotAfter.ToString('yyyy-MM-dd') }) -join '; ')"
|
||||
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') {
|
||||
try { $t = New-Object Net.Sockets.TcpClient($h,636); $script:err = $null
|
||||
$s = New-Object Net.Security.SslStream($t.GetStream(), $false, { param($a,$b,$c,$e) $script:err = $e; $true })
|
||||
$s.AuthenticateAsClient($h); "tls $h policyErrors=$script:err proto=$($s.SslProtocol)"; $s.Dispose(); $t.Dispose() } catch { "tls $h FAILED $($_.Exception.Message)" } }
|
||||
|
||||
# Local state: greenfield expected
|
||||
foreach ($d in 'E:\ApiInstall','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\ScadaBridge','C:\ProgramData\ZB\kek') { "path $d exists=$(Test-Path $d)" }
|
||||
Get-Service | Where-Object { $_.Name -match 'MxAccess|MxGateway|ScadaBridge|ZbOtel|OtOpcUa|Historian' } | ForEach-Object { "service $($_.Name) $($_.Status)" }
|
||||
$listen = Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue | Where-Object { $_.LocalPort -in 4317,5120,5130,8081,8082,8083,8084,8085,8889 }
|
||||
foreach ($l in $listen) { "PORT IN USE $($l.LocalAddress):$($l.LocalPort) pid $($l.OwningProcess) $((Get-Process -Id $l.OwningProcess -ErrorAction SilentlyContinue).ProcessName)" }
|
||||
if (-not $listen) { 'ports 4317 5120 5130 8081-8085 8889: all free' }
|
||||
$fw = Get-NetFirewallProfile -ErrorAction SilentlyContinue | ForEach-Object { "$($_.Name)=$($_.Enabled)" }
|
||||
"windows firewall profiles: $($fw -join ' ')"
|
||||
|
||||
# Name resolution + paths to the other boxes (plan S4). open = listener; refused = path open, nothing listening yet;
|
||||
# timeout = probably filtered. Nothing listens yet on a greenfield box, so 'refused' is the good answer here.
|
||||
function Probe($h, $p) {
|
||||
$c = New-Object Net.Sockets.TcpClient
|
||||
try { $ok = $c.ConnectAsync($h, $p).Wait(3000); if ($ok) { 'open' } else { 'timeout' } }
|
||||
catch { $m = $_.Exception.InnerException; while ($m.InnerException) { $m = $m.InnerException }; if ($m.Message -match 'refused|actively') { 'refused' } else { "error: $($m.Message)" } }
|
||||
finally { $c.Dispose() } }
|
||||
foreach ($n in 'wonder-app-vp03','wonder-app-vp04','wonder-par-vp03','wonder-par-vp04') {
|
||||
$f = "$n.zmr.zimmer.com"; try { $a = ([Net.Dns]::GetHostAddresses($f) | Select-Object -First 1).IPAddressToString } catch { $a = 'NXDOMAIN' }
|
||||
"dns $f -> $a (plan $($ip[$n.ToUpperInvariant()]))" }
|
||||
$targets = if ($central) {
|
||||
@(@{h = $ip[($(if ($box -eq 'WONDER-APP-VP03') {'WONDER-APP-VP04'} else {'WONDER-APP-VP03'}))]; p = 8081, 8085, 8083},
|
||||
@{h = '10.201.75.86'; p = 8083, 5120}, @{h = '10.201.75.87'; p = 8083, 5120})
|
||||
} else {
|
||||
@(@{h = $ip[($(if ($box -eq 'WONDER-PAR-VP03') {'WONDER-PAR-VP04'} else {'WONDER-PAR-VP03'}))]; p = 8082, 8083, 5120},
|
||||
@{h = '10.220.153.205'; p = 8083, 5120}, @{h = '10.220.153.210'; p = 8083, 5120})
|
||||
}
|
||||
foreach ($t in $targets) { foreach ($p in $t.p) { "path -> $($t.h):$p $(Probe $t.h $p)" } }
|
||||
if ($SqlHost) { "path -> SQL $SqlHost`:1433 $(Probe $SqlHost 1433)" }
|
||||
if ($GrHost) { "path -> GR $GrHost`:1433 $(Probe $GrHost 1433)" }
|
||||
|
||||
# Staged artifacts
|
||||
foreach ($sums in Get-ChildItem 'E:\psrem-stage' -Recurse -Filter 'SHA256SUMS.txt' -ErrorAction SilentlyContinue) {
|
||||
foreach ($line in Get-Content $sums.FullName) { if ($line -match '^([0-9a-fA-F]{64})\s+\*?(.+)$') {
|
||||
$f = Join-Path $sums.DirectoryName $Matches[2]
|
||||
if (-not (Test-Path $f)) { "stage MISSING $f" } elseif ((Get-FileHash $f).Hash -ne $Matches[1].ToUpperInvariant()) { "stage HASH MISMATCH $f" } else { "stage ok $f" } } } }
|
||||
foreach ($e in 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe','E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe') { if (Test-Path $e) { "tool $e sha256 $((Get-FileHash $e).Hash.Substring(0,12))" } }
|
||||
"PREFLIGHT DONE"
|
||||
@@ -0,0 +1,56 @@
|
||||
# Seed nam-ldap-svc-password into this PROD box's stores BEFORE each service's first start (plan S3A.4). QA seed-secret.ps1
|
||||
# retargeted. Each store's location and KEK come from the installed (stopped) service's own env, never assumed.
|
||||
# -Target MxAccessGw : this box's mxgw SQLite store (all four boxes)
|
||||
# -Target ScadaBridge-Central : the SHARED SQL Server store (zbsecrets), ONCE, on vp03 only, with the SQL-aware CLI
|
||||
# (the headless 0.6.6 CLI writes SQLite only). Needs sql\zbsecrets-schema-v1.sql applied.
|
||||
# The value arrives as -ValueFile over SMB (Keychain -> 0600 temp -> SMB), is ACL'd, read back, compared, then DELETED.
|
||||
# It is never printed and never on argv.
|
||||
param(
|
||||
[Parameter(Mandatory)][ValidateSet('MxAccessGw','ScadaBridge-Central')][string]$Target,
|
||||
[Parameter(Mandatory)][string]$ValueFile
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$cli = 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe'
|
||||
$cliSql = 'E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe'
|
||||
$name = 'nam-ldap-svc-password'
|
||||
$desc = 'NAM LDAP lookup account (svc_igntadmin, PROD interim D-P5)'
|
||||
& icacls.exe $ValueFile /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
|
||||
function SvcEnv($svc) {
|
||||
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"
|
||||
if (-not (Test-Path $k)) { throw "service $svc is not installed" }
|
||||
$l = @((Get-ItemProperty $k -ErrorAction SilentlyContinue).Environment) + @((Get-ItemProperty "$k\Parameters" -ErrorAction SilentlyContinue).AppEnvironmentExtra) | Where-Object { $_ }
|
||||
$h = @{}; foreach ($e in $l) { $i = $e.IndexOf('='); if ($i -gt 0) { $h[$e.Substring(0,$i)] = $e.Substring($i+1) } }; $h
|
||||
}
|
||||
$names = 'Secrets__SqlitePath','Secrets__MasterKey__Source','Secrets__MasterKey__FilePath','Secrets__MasterKey__EnvVarName','Secrets__Store__Provider','Secrets__SqlServer__ConnectionString','ZB_SECRETS_MASTER_KEY'
|
||||
$fail = 0
|
||||
try {
|
||||
if ($Target -eq 'ScadaBridge-Central' -and $env:COMPUTERNAME -ne 'WONDER-APP-VP03') { throw 'the shared central store is seeded once, from vp03' }
|
||||
$svc = Get-Service $Target; if ($svc.Status -ne 'Stopped') { throw "$Target is $($svc.Status); seed before the first start" }
|
||||
$want = [IO.File]::ReadAllText($ValueFile).TrimEnd("`r", "`n")
|
||||
if (-not $want) { throw 'value file is empty' }
|
||||
"value file read, length $($want.Length)"
|
||||
$e = SvcEnv $Target
|
||||
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue; if ($e.ContainsKey($n)) { Set-Item "Env:$n" $e[$n] } }
|
||||
$exe = $cli
|
||||
if ($Target -eq 'ScadaBridge-Central') {
|
||||
$env:Secrets__Store__Provider = 'SqlServer'; $exe = $cliSql
|
||||
if (-not $env:Secrets__SqlServer__ConnectionString) { throw 'Central env has no Secrets__SqlServer__ConnectionString' }
|
||||
}
|
||||
if ($env:Secrets__MasterKey__Source -ne 'File') { throw "master key source '$($env:Secrets__MasterKey__Source)' is not File" }
|
||||
if (-not (Test-Path $env:Secrets__MasterKey__FilePath)) { throw "KEK $($env:Secrets__MasterKey__FilePath) missing" }
|
||||
$where = if ($Target -eq 'ScadaBridge-Central') { 'SqlServer shared store' } else { $env:Secrets__SqlitePath }
|
||||
$tmp = "$ValueFile.nl"; [IO.File]::WriteAllText($tmp, $want)
|
||||
& icacls.exe $tmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
|
||||
$out = & cmd.exe /c "`"$exe`" set $name - --description `"$desc`" < `"$tmp`"" 2>&1
|
||||
$rc = $LASTEXITCODE
|
||||
Remove-Item $tmp -Force -ErrorAction SilentlyContinue
|
||||
$got = (& $exe get $name 2>$null | Out-String).TrimEnd("`r", "`n")
|
||||
$match = ($got -ceq $want)
|
||||
if (-not $match) { try { $j = $got | ConvertFrom-Json; $match = ($j.value -ceq $want) } catch {} }
|
||||
"$Target -> $where kek=$($env:Secrets__MasterKey__FilePath) kek-sha256=$((Get-FileHash $env:Secrets__MasterKey__FilePath).Hash.Substring(0,8)) set rc=$rc out=$(($out | Out-String).Trim()) readback-match=$match"
|
||||
if ($rc -ne 0 -or -not $match) { $fail++ }
|
||||
} finally {
|
||||
Remove-Item $ValueFile, "$ValueFile.nl" -Force -ErrorAction SilentlyContinue; "value file removed: $(-not (Test-Path $ValueFile))"
|
||||
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue }
|
||||
}
|
||||
"SEED DONE failures=$fail"; exit $fail
|
||||
@@ -0,0 +1,213 @@
|
||||
#requires -Version 5.1
|
||||
# PROD STAGE COPY (2026-11-19): mxaccessgw e74918cc scripts/deploy/Install-MxGatewayService.ps1 + 3 patches marked PROD-STAGE PATCH.
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box NEW-BOX install of the MxAccess Gateway as an NSSM service (mxaccessgw#220).
|
||||
Companion of Upgrade-MxGatewayService.ps1; see docs/runbooks/DeployUpgradeNssmService.md.
|
||||
|
||||
.DESCRIPTION
|
||||
RUN ON THE WINDOWS BOX (Windows PowerShell 5.1). Creates the service ONCE, with:
|
||||
* Application / AppDirectory / DisplayName / Description, auto start, account (default LocalSystem);
|
||||
* NSSM exit handling: restart on any exit with a throttle, graceful-stop timeouts that cover the
|
||||
Worker shutdown grace (MxGateway:Worker:ShutdownTimeoutSeconds, default 10 s);
|
||||
* stdout/stderr redirection with rotation;
|
||||
* Windows SCM recovery (sc failure: restart, restart, restart + reset period);
|
||||
* the environment block, set in ONE `nssm set AppEnvironmentExtra` call on a service that has none
|
||||
(so there is nothing to wipe). The env comes from -EnvFile (KEY=VALUE lines, # comments) and/or
|
||||
-Env (hashtable); -Env wins. Secret VALUES are never printed (dry-run shows names and masks values).
|
||||
It does NOT start the service unless -Start is given (review the env first), and it refuses to run if
|
||||
the service already exists (use Upgrade-MxGatewayService.ps1) or if any gateway process is running
|
||||
(OrphanWorkerTerminator trap: a second gateway kills live Workers).
|
||||
|
||||
Required env (checked by name): ZB_SECRETS_MASTER_KEY, a *Pepper* entry (MxGateway__ApiKeyPepper),
|
||||
MxGateway__Worker__ExecutablePath (defaults to <WorkerDir>\ZB.MOM.WW.MxGateway.Worker.exe),
|
||||
MxGateway__Authentication__SqlitePath and Secrets__SqlitePath (default under -DataDir; must be absolute and
|
||||
OUTSIDE the content root, the validator is fail-closed), ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT.
|
||||
Kestrel endpoints default to docs/GatewayConfiguration.md's cleartext h2c :5120 + dashboard :5130 unless
|
||||
the env supplies Kestrel__Endpoints__*.
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none.
|
||||
#>
|
||||
[CmdletBinding(SupportsShouldProcess = $true)]
|
||||
param(
|
||||
[Parameter(Mandatory = $true)][string]$ServerDir,
|
||||
[Parameter(Mandatory = $true)][string]$WorkerDir,
|
||||
[string]$ServiceName = 'MxAccessGw',
|
||||
[string]$NssmPath = 'nssm',
|
||||
[string]$DataDir = 'C:\ProgramData\MxGateway',
|
||||
[string]$LogDir,
|
||||
[string]$EnvFile,
|
||||
[hashtable]$Env = @{},
|
||||
[string]$ServerExeName = 'ZB.MOM.WW.MxGateway.Server.exe',
|
||||
# PROD-STAGE PATCH 1: the wonder Worker build is named MxGateway.Worker.exe, not ZB.MOM.WW.MxGateway.Worker.exe.
|
||||
[string]$WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe',
|
||||
[string]$ObjectName = 'LocalSystem',
|
||||
[string]$ServiceAccountPassword,
|
||||
[string]$EnvironmentName = 'Production',
|
||||
[int]$RestartDelayMs = 5000,
|
||||
[int]$ThrottleMs = 10000,
|
||||
[int]$StopGraceMs = 20000,
|
||||
[int]$RotateBytes = 10485760,
|
||||
[switch]$Start,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
$ServerProc = 'ZB.MOM.WW.MxGateway.Server'
|
||||
$WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
|
||||
|
||||
function Say([string]$m) { Write-Host $m }
|
||||
function Fail([string]$m) { throw "ABORT: $m" }
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
|
||||
Say "[ACTION] $Description"; & $Action
|
||||
}
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
$raw = & $NssmPath @NssmArgs 2>&1; $code = $LASTEXITCODE
|
||||
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
|
||||
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs[0..1] -join ' ') failed (exit $code): $text" }
|
||||
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
|
||||
}
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
function Mask([string]$k, [string]$v) {
|
||||
# PROD-STAGE PATCH 3: connection strings can carry a SQL password; mask them too.
|
||||
if ($k -match 'KEY|PASSWORD|SECRET|PEPPER|TOKEN|CONNECTIONSTRING' -and $k -notmatch 'Path$|Url$') { '****' } else { $v }
|
||||
}
|
||||
|
||||
try {
|
||||
Say "Install-MxGatewayService service=$ServiceName dry=$($script:Dry)"
|
||||
|
||||
# --- environment assembly -------------------------------------------------------------------------
|
||||
$envMap = [ordered]@{}
|
||||
if ($EnvFile) {
|
||||
if (-not (Test-Path -LiteralPath $EnvFile)) { Fail "-EnvFile '$EnvFile' not found" }
|
||||
foreach ($line in (Get-Content -LiteralPath $EnvFile)) {
|
||||
$t = $line.Trim()
|
||||
if (-not $t -or $t.StartsWith('#')) { continue }
|
||||
if ($t -match '^([^=\s]+)=(.*)$') { $envMap[$Matches[1]] = $Matches[2] } else { Fail "-EnvFile line is not KEY=VALUE: '$t'" }
|
||||
}
|
||||
}
|
||||
foreach ($k in $Env.Keys) { $envMap[[string]$k] = [string]$Env[$k] }
|
||||
|
||||
$workerExe = Join-Path $WorkerDir $WorkerExeName
|
||||
if (-not $envMap.Contains('MxGateway__Worker__ExecutablePath')) { $envMap['MxGateway__Worker__ExecutablePath'] = $workerExe }
|
||||
if (-not $envMap.Contains('MxGateway__Authentication__SqlitePath')) { $envMap['MxGateway__Authentication__SqlitePath'] = Join-Path $DataDir 'gateway-auth.db' }
|
||||
if (-not $envMap.Contains('Secrets__SqlitePath')) { $envMap['Secrets__SqlitePath'] = Join-Path $DataDir 'mxgateway-secrets.db' }
|
||||
if (-not $envMap.Contains('ASPNETCORE_ENVIRONMENT')) { $envMap['ASPNETCORE_ENVIRONMENT'] = $EnvironmentName }
|
||||
if (-not $envMap.Contains('DOTNET_ENVIRONMENT')) { $envMap['DOTNET_ENVIRONMENT'] = $EnvironmentName }
|
||||
if (-not ($envMap.Keys | Where-Object { $_ -like 'Kestrel__Endpoints__*' })) {
|
||||
$envMap['Kestrel__Endpoints__Http__Url'] = 'http://0.0.0.0:5120'
|
||||
$envMap['Kestrel__Endpoints__Http__Protocols'] = 'Http2'
|
||||
$envMap['Kestrel__Endpoints__Dashboard__Url'] = 'http://0.0.0.0:5130'
|
||||
Say '[info] no Kestrel__Endpoints__* supplied: using the documented cleartext defaults (5120 h2c, 5130 dashboard)'
|
||||
}
|
||||
|
||||
# --- validation -------------------------------------------------------------------------------------
|
||||
# PROD-STAGE PATCH 2: accept a file KEK (Secrets__MasterKey__Source=File + FilePath), the QA/PROD pattern.
|
||||
if ($envMap['Secrets__MasterKey__Source'] -eq 'File') {
|
||||
$kf = $envMap['Secrets__MasterKey__FilePath']
|
||||
if (-not $kf -or -not (Test-Rooted $kf)) { Fail 'Secrets__MasterKey__Source=File needs an absolute Secrets__MasterKey__FilePath' }
|
||||
if (-not $script:Dry -and -not (Test-Path -LiteralPath $kf)) { Fail "KEK file '$kf' not found" }
|
||||
} elseif (-not $envMap.Contains('ZB_SECRETS_MASTER_KEY') -or -not $envMap['ZB_SECRETS_MASTER_KEY']) { Fail 'env must supply ZB_SECRETS_MASTER_KEY (the Secrets KEK) or Secrets__MasterKey__Source=File' }
|
||||
if (-not ($envMap.Keys | Where-Object { $_ -match 'Pepper' })) { Fail 'env must supply the API-key pepper (MxGateway__ApiKeyPepper); without it no issued token verifies' }
|
||||
foreach ($k in 'MxGateway__Authentication__SqlitePath', 'Secrets__SqlitePath') {
|
||||
$v = $envMap[$k]
|
||||
if (-not (Test-Rooted $v)) { Fail "$k '$v' must be absolute" }
|
||||
if ((Test-Inside $v $ServerDir) -or (Test-Inside $v $WorkerDir)) { Fail "$k '$v' is inside the content root; the validator is fail-closed and an upgrade swap would lose it" }
|
||||
}
|
||||
foreach ($d in @($ServerDir, $WorkerDir)) { if (-not (Test-Rooted $d)) { Fail "'$d' must be an absolute path" } }
|
||||
if ((Test-Inside $WorkerDir $ServerDir) -or (Test-Inside $ServerDir $WorkerDir)) { Fail 'Server and Worker directories must be siblings, not nested (the upgrade swap renames them independently)' }
|
||||
if (-not (Test-Path -LiteralPath (Join-Path $ServerDir $ServerExeName))) { Fail "$ServerExeName not found in $ServerDir" }
|
||||
if (-not (Test-Path -LiteralPath $workerExe)) { Fail "$WorkerExeName not found in $WorkerDir" }
|
||||
if ($envMap['MxGateway__Worker__ExecutablePath'] -ne $workerExe) { Say '[WARN] MxGateway__Worker__ExecutablePath differs from <WorkerDir>\Worker.exe; keep ONE canonical spelling' }
|
||||
if ($envMap['ASPNETCORE_ENVIRONMENT'] -ne 'Production') { Say '[WARN] ASPNETCORE_ENVIRONMENT is not exactly Production: DisableLogin / plaintext-LDAP hard-stops are only enforced for Production' }
|
||||
|
||||
# --- one-gateway guard + existence -------------------------------------------------------------------
|
||||
$st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
|
||||
if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" }
|
||||
# Both naming schemes (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes) plus the exe being installed (#230).
|
||||
$gwImages = @($ServerProc, $WorkerProc, 'MxGateway.Server', 'MxGateway.Worker', [IO.Path]::GetFileNameWithoutExtension($ServerExeName)) | Select-Object -Unique
|
||||
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in $gwImages })
|
||||
if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" }
|
||||
|
||||
if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' }
|
||||
$exe = Join-Path $ServerDir $ServerExeName
|
||||
|
||||
Say "[info] env entries ($($envMap.Count)):"
|
||||
foreach ($k in $envMap.Keys) { Say " $k=$(Mask $k $envMap[$k])" }
|
||||
|
||||
# --- install ------------------------------------------------------------------------------------------------
|
||||
Step "create $DataDir and $LogDir (stores/certs/logs live OUTSIDE the content root)" {
|
||||
New-Item -ItemType Directory -Force -Path $DataDir, $LogDir | Out-Null
|
||||
}
|
||||
Step "nssm install $ServiceName `"$exe`"" { [void](Invoke-Nssm -NssmArgs @('install', $ServiceName, $exe)); $script:Installed = $true }
|
||||
$settings = @(
|
||||
@('AppDirectory', $ServerDir),
|
||||
@('DisplayName', 'MxAccess Gateway'),
|
||||
@('Description', 'MxAccess Gateway (gRPC) - Server; launches the x86 Worker per session'),
|
||||
@('Start', 'SERVICE_AUTO_START'),
|
||||
@('Type', 'SERVICE_WIN32_OWN_PROCESS'),
|
||||
@('AppExit', 'Default', 'Restart'),
|
||||
@('AppRestartDelay', "$RestartDelayMs"),
|
||||
@('AppThrottle', "$ThrottleMs"),
|
||||
@('AppStopMethodConsole', "$StopGraceMs"),
|
||||
@('AppStopMethodWindow', '0'),
|
||||
@('AppStopMethodThreads', '0'),
|
||||
@('AppStdout', (Join-Path $LogDir 'service-stdout.log')),
|
||||
@('AppStderr', (Join-Path $LogDir 'service-stderr.log')),
|
||||
@('AppRotateFiles', '1'),
|
||||
@('AppRotateOnline', '1'),
|
||||
@('AppRotateBytes', "$RotateBytes")
|
||||
)
|
||||
foreach ($s in $settings) {
|
||||
Step "nssm set $ServiceName $($s -join ' ')" { [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName) + $s)) }
|
||||
}
|
||||
if ($ObjectName -and $ObjectName -ne 'LocalSystem') {
|
||||
Step "nssm set $ServiceName ObjectName $ObjectName <password>" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', $ObjectName, $ServiceAccountPassword)) }
|
||||
} else {
|
||||
Step "nssm set $ServiceName ObjectName LocalSystem" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', 'LocalSystem')) }
|
||||
}
|
||||
|
||||
# Env block: first and only AppEnvironmentExtra write; the service has none, so nothing is wiped.
|
||||
$pairs = @($envMap.Keys | ForEach-Object { "$_=$($envMap[$_])" })
|
||||
Step "nssm set $ServiceName AppEnvironmentExtra <$($envMap.Count) entries, values hidden>" {
|
||||
[void](Invoke-Nssm -NssmArgs (@('set', $ServiceName, 'AppEnvironmentExtra') + $pairs))
|
||||
}
|
||||
|
||||
# SCM recovery: restart x3 (5 s, 5 s, 30 s), reset the counter after 1 day; also on non-crash exits.
|
||||
Step "sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/30000" {
|
||||
& sc.exe failure $ServiceName reset= 86400 actions= 'restart/5000/restart/5000/restart/30000' | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failure exited $LASTEXITCODE" }
|
||||
}
|
||||
Step "sc.exe failureflag $ServiceName 1" {
|
||||
& sc.exe failureflag $ServiceName 1 | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failureflag exited $LASTEXITCODE" }
|
||||
}
|
||||
|
||||
# Verify read-back (names only) so a silent miss is caught before the first start.
|
||||
Step "verify: nssm get AppEnvironmentExtra has $($envMap.Count) entries" {
|
||||
$txt = (Invoke-Nssm -NssmArgs @('get', $ServiceName, 'AppEnvironmentExtra')).Text
|
||||
$n = @($txt -split "[`r`n]+" | Where-Object { $_.Trim() -match '^[^=\s]+=' }).Count
|
||||
if ($n -ne $envMap.Count) { Fail "env read-back has $n entries, expected $($envMap.Count)" }
|
||||
}
|
||||
|
||||
if ($Start) {
|
||||
Step "nssm start $ServiceName (exactly once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) }
|
||||
Say 'Next: GET /health/ready on the dashboard port, then prove a Worker session with an API key (runbook section 7).'
|
||||
} else {
|
||||
Say '[info] service installed STOPPED. Review the env, then `nssm start` (or re-run with -Start). Seed secrets with the `secret` CLI using the SAME Secrets__SqlitePath + ZB_SECRETS_MASTER_KEY.'
|
||||
}
|
||||
Say '== DONE =='
|
||||
exit 0
|
||||
} catch {
|
||||
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
|
||||
if ($script:Installed) { Write-Host "PARTIAL INSTALL: inspect, then `nssm remove $ServiceName confirm` and re-run." -ForegroundColor Yellow }
|
||||
exit 1
|
||||
}
|
||||
@@ -0,0 +1,869 @@
|
||||
#requires -Version 5.1
|
||||
<#
|
||||
.SYNOPSIS
|
||||
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
|
||||
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
|
||||
|
||||
.DESCRIPTION
|
||||
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
|
||||
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
|
||||
|
||||
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
|
||||
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
|
||||
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
|
||||
restore -> start -> verify.
|
||||
|
||||
Safety properties:
|
||||
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
|
||||
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
|
||||
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
|
||||
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
|
||||
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
|
||||
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
|
||||
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
|
||||
never printed) at preflight, after the stop, after the swap, after the start and after the stability
|
||||
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
|
||||
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
|
||||
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
|
||||
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
|
||||
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
|
||||
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
|
||||
|
||||
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
|
||||
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
|
||||
|
||||
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
|
||||
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
|
||||
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
|
||||
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
|
||||
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
|
||||
.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath.
|
||||
.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify)
|
||||
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
|
||||
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
|
||||
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
|
||||
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
|
||||
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
|
||||
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
|
||||
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
|
||||
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
|
||||
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
|
||||
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
|
||||
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
|
||||
#>
|
||||
[CmdletBinding(SupportsShouldProcess = $true)]
|
||||
param(
|
||||
[string]$ServerSource,
|
||||
[string]$WorkerSource,
|
||||
[string]$ServiceName = 'MxAccessGw',
|
||||
[string]$NssmPath = 'nssm',
|
||||
[string]$ServerDir,
|
||||
[string]$WorkerDir,
|
||||
[string]$ExpectedSha,
|
||||
[string]$Label,
|
||||
[string]$HealthBaseUrl = 'http://localhost:5130',
|
||||
[switch]$Rollback,
|
||||
[string]$BackupStamp,
|
||||
[switch]$RestoreStores,
|
||||
[switch]$NoAutoRollback,
|
||||
[switch]$TakeNewAppSettings,
|
||||
[switch]$AllowRuntimeDowngrade,
|
||||
[switch]$SkipFreeSpaceCheck,
|
||||
[double]$MinFreeGB = 2,
|
||||
[int]$StopTimeoutSeconds = 30,
|
||||
[int]$StableSeconds = 60,
|
||||
[int]$ReadyTimeoutSeconds = 120,
|
||||
[switch]$DryRun
|
||||
)
|
||||
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
|
||||
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
|
||||
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
|
||||
$WhatIfPreference = $false
|
||||
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
|
||||
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
|
||||
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
|
||||
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
|
||||
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
|
||||
$script:ServerImages = $script:KnownServerImages
|
||||
$script:WorkerImages = $script:KnownWorkerImages
|
||||
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
|
||||
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
|
||||
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
|
||||
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
|
||||
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
|
||||
$script:Notes = New-Object System.Collections.Generic.List[string]
|
||||
$script:Redact = New-Object System.Collections.Generic.List[string]
|
||||
$script:LockStream = $null
|
||||
$script:BackupDir = $null
|
||||
$script:EnvWarnOnly = $false
|
||||
|
||||
# --- output (secret values are redacted from every line) ----------------------------------------------------
|
||||
|
||||
function Hide([string]$m) {
|
||||
if ($null -eq $m) { return '' }
|
||||
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
|
||||
$m
|
||||
}
|
||||
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
|
||||
function Info([string]$m) { Say "[info] $m" }
|
||||
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
|
||||
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
|
||||
|
||||
# Mutating step: print in dry-run, run otherwise.
|
||||
function Step([string]$Description, [scriptblock]$Action) {
|
||||
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
|
||||
Say "[ACTION] $Description"
|
||||
& $Action
|
||||
}
|
||||
|
||||
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
|
||||
|
||||
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
|
||||
|
||||
function Invoke-Nssm {
|
||||
param([string[]]$NssmArgs, [switch]$AllowFail)
|
||||
$raw = & $NssmPath @NssmArgs 2>&1
|
||||
$code = $LASTEXITCODE
|
||||
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
|
||||
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" }
|
||||
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
|
||||
}
|
||||
|
||||
function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text }
|
||||
|
||||
function Get-ServiceState {
|
||||
$r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
|
||||
if ($r.Code -ne 0) { return 'MISSING' }
|
||||
($r.Text -split "`n")[0].Trim()
|
||||
}
|
||||
|
||||
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
|
||||
function Read-EnvBlockLines {
|
||||
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
|
||||
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
|
||||
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
|
||||
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
|
||||
return @()
|
||||
}
|
||||
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
|
||||
}
|
||||
|
||||
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
|
||||
function Get-ProcessTable {
|
||||
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
|
||||
return @(Get-CimInstance Win32_Process | ForEach-Object {
|
||||
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
|
||||
})
|
||||
}
|
||||
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
|
||||
$p = $null; try { $p = $_.Path } catch { }
|
||||
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
|
||||
})
|
||||
}
|
||||
|
||||
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
|
||||
function Get-ServiceHostPid {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
|
||||
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
|
||||
$null
|
||||
}
|
||||
|
||||
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
|
||||
function Get-OtherGatewayServices([string]$ExeName) {
|
||||
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
|
||||
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
|
||||
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
|
||||
}
|
||||
|
||||
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
|
||||
|
||||
function Get-HttpStatus([string]$Url) {
|
||||
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
|
||||
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
|
||||
}
|
||||
|
||||
function Get-ProductSha([string]$Exe) {
|
||||
try {
|
||||
$pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion
|
||||
if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() }
|
||||
} catch { }
|
||||
$null
|
||||
}
|
||||
|
||||
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
|
||||
function Get-ExeArchitecture([string]$Exe) {
|
||||
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
|
||||
}
|
||||
|
||||
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
|
||||
function Protect-Directory([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
|
||||
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
|
||||
}
|
||||
|
||||
function Export-ServiceKey([string]$Destination) {
|
||||
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
|
||||
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
|
||||
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
|
||||
}
|
||||
|
||||
function Get-DirSddl([string]$Path) {
|
||||
if (-not (Test-IsWindowsHost)) { return $null }
|
||||
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
|
||||
}
|
||||
|
||||
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
|
||||
function Copy-ExplicitAcl([string]$From, [string]$To) {
|
||||
if (-not (Test-IsWindowsHost)) { return }
|
||||
try {
|
||||
$a = Get-Acl -LiteralPath $From
|
||||
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
|
||||
Set-Acl -LiteralPath $To -AclObject $a
|
||||
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
|
||||
}
|
||||
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
# --- pure helpers ---------------------------------------------------------------------------------------------
|
||||
|
||||
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
|
||||
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
|
||||
function Test-Inside([string]$Path, [string]$Root) {
|
||||
if (-not $Path -or -not $Root) { return $false }
|
||||
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
|
||||
$a -eq $b -or $a.StartsWith("$b/")
|
||||
}
|
||||
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
|
||||
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
|
||||
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
|
||||
|
||||
function Test-ShaMatch([string]$Actual, [string]$Expected) {
|
||||
if (-not $Actual -or -not $Expected) { return $false }
|
||||
$e = $Expected.ToLowerInvariant()
|
||||
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
|
||||
}
|
||||
|
||||
function Get-TextHashPrefix([string]$Text) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create()
|
||||
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
|
||||
finally { $sha.Dispose() }
|
||||
}
|
||||
function Get-FileSha([string]$Path) {
|
||||
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
|
||||
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
|
||||
}
|
||||
|
||||
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
|
||||
|
||||
# name -> value. Every sensitive value is registered for redaction.
|
||||
function ConvertTo-EnvMap([string[]]$Lines) {
|
||||
$map = [ordered]@{}
|
||||
foreach ($line in $Lines) {
|
||||
$i = $line.IndexOf('=')
|
||||
if ($i -le 0) { continue }
|
||||
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
|
||||
$map[$k] = $v
|
||||
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
|
||||
}
|
||||
$map
|
||||
}
|
||||
|
||||
# Names + per-value SHA-256 prefixes. Never carries a value.
|
||||
function Get-EnvFingerprint([string[]]$Lines) {
|
||||
$map = ConvertTo-EnvMap $Lines
|
||||
$entries = [ordered]@{}
|
||||
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
|
||||
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
|
||||
}
|
||||
|
||||
# List of differences (names only). Empty when identical.
|
||||
function Compare-EnvFingerprint($Before, $After) {
|
||||
$d = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($k in $Before.Entries.Keys) {
|
||||
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
|
||||
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
|
||||
}
|
||||
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
|
||||
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
|
||||
, $d.ToArray()
|
||||
}
|
||||
|
||||
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
|
||||
|
||||
# Re-read the env block and abort on any difference from the preflight fingerprint.
|
||||
function Assert-EnvUnchanged($ctx, [string]$When) {
|
||||
$now = Get-EnvFingerprint (Read-EnvBlockLines)
|
||||
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
|
||||
if ($diff.Count -gt 0) {
|
||||
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
|
||||
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
|
||||
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
|
||||
}
|
||||
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
|
||||
}
|
||||
|
||||
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
|
||||
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
|
||||
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
|
||||
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
|
||||
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
|
||||
}
|
||||
|
||||
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
|
||||
|
||||
function Get-DirSize([string]$p) {
|
||||
if (-not (Test-Path -LiteralPath $p)) { return 0 }
|
||||
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
|
||||
if ($s) { [double]$s } else { 0 }
|
||||
}
|
||||
|
||||
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
|
||||
function Get-BundledRuntimeVersion([string]$Dir) {
|
||||
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
|
||||
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
|
||||
$null
|
||||
}
|
||||
|
||||
# --- processes --------------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
|
||||
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
|
||||
|
||||
function Get-GatewayProcesses($ctx) {
|
||||
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
|
||||
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
|
||||
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
|
||||
})
|
||||
}
|
||||
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
|
||||
|
||||
function Assert-NoGatewayProcess($ctx, [string]$When) {
|
||||
$left = @(Get-GatewayProcesses $ctx)
|
||||
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
|
||||
}
|
||||
|
||||
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
|
||||
function Assert-SingleGateway($ctx, [string]$State) {
|
||||
$procs = @(Get-GatewayProcesses $ctx)
|
||||
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
|
||||
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
|
||||
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
|
||||
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
|
||||
switch ($State) {
|
||||
'SERVICE_STOPPED' {
|
||||
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
|
||||
}
|
||||
'SERVICE_RUNNING' {
|
||||
if ($servers.Count -eq 0) {
|
||||
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
|
||||
}
|
||||
$s = $servers[0]
|
||||
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
|
||||
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
|
||||
$hostPid = Get-ServiceHostPid
|
||||
if ($hostPid) {
|
||||
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
|
||||
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
|
||||
} else { Warn 'service host PID unavailable; parent check skipped' }
|
||||
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
|
||||
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
|
||||
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
|
||||
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
|
||||
}
|
||||
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
|
||||
default {
|
||||
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# --- context discovery (read only) -------------------------------------------------------------------------
|
||||
|
||||
function Get-Context {
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" }
|
||||
$ctx = [ordered]@{}
|
||||
$ctx.State = $state
|
||||
$ctx.Application = Get-NssmValue 'Application'
|
||||
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
|
||||
$lines = @(Read-EnvBlockLines)
|
||||
$ctx.Env = ConvertTo-EnvMap $lines
|
||||
$ctx.EnvFp = Get-EnvFingerprint $lines
|
||||
$ctx.EnvCount = $ctx.EnvFp.Count
|
||||
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
|
||||
$wexe = $ctx.Env[$script:EnvWorkerExe]
|
||||
$ctx.WorkerExe = $wexe
|
||||
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
|
||||
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
|
||||
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
|
||||
$pd = 'C:\ProgramData\MxGateway'
|
||||
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
|
||||
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
|
||||
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
|
||||
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
|
||||
$ctx.ServerExeName = Get-Leaf $ctx.Application
|
||||
$ctx.WorkerExeName = Get-Leaf $wexe
|
||||
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
|
||||
# Process image names: the installed ones first, then both known schemes.
|
||||
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
|
||||
[pscustomobject]$ctx
|
||||
}
|
||||
|
||||
# --- preflight -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Test-MasterKey($ctx) {
|
||||
$src = $ctx.Env['Secrets__MasterKey__Source']
|
||||
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
|
||||
if ($ctx.KekFile -or $src -eq 'File') {
|
||||
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
|
||||
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
|
||||
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
|
||||
} elseif ($ctx.Env.Contains($envVar)) {
|
||||
Info "master key: env block entry $envVar"
|
||||
} else {
|
||||
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
|
||||
}
|
||||
}
|
||||
|
||||
function Invoke-Preflight($ctx) {
|
||||
Say '== Preflight =='
|
||||
Info "service '$ServiceName' state: $($ctx.State)"
|
||||
Info "Application : $($ctx.Application)"
|
||||
Info "AppDirectory: $($ctx.AppDirectory)"
|
||||
Info "Server dir : $($ctx.ServerDir)"
|
||||
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
|
||||
Info "API-key DB : $($ctx.AuthDb)"
|
||||
Info "Secrets DB : $($ctx.SecretsDb)"
|
||||
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
|
||||
|
||||
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
|
||||
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
|
||||
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
|
||||
}
|
||||
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
|
||||
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
|
||||
}
|
||||
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
|
||||
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
|
||||
|
||||
# State that must survive: rooted and outside the swapped directories.
|
||||
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
|
||||
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
|
||||
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." }
|
||||
}
|
||||
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
|
||||
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
|
||||
}
|
||||
Test-MasterKey $ctx
|
||||
|
||||
# Directories
|
||||
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
|
||||
if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" }
|
||||
}
|
||||
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
|
||||
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
|
||||
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
|
||||
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
|
||||
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
|
||||
|
||||
# One-gateway guard (fail closed)
|
||||
Assert-SingleGateway $ctx $ctx.State
|
||||
$others = Get-OtherGatewayServices $ctx.ServerExeName
|
||||
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
|
||||
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
|
||||
else { Info "no other service launches $($ctx.ServerExeName)" }
|
||||
|
||||
if ($Rollback) { Say '== Preflight OK =='; return }
|
||||
|
||||
# Sources
|
||||
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
|
||||
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
|
||||
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
|
||||
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
|
||||
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
|
||||
}
|
||||
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
|
||||
$arch = Get-ExeArchitecture $stagedWorkerExe
|
||||
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
|
||||
Info "staged Worker architecture: X86"
|
||||
$sSha = Get-ProductSha $stagedServerExe
|
||||
$wSha = Get-ProductSha $stagedWorkerExe
|
||||
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
|
||||
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
|
||||
if ($ExpectedSha) {
|
||||
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
|
||||
if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" }
|
||||
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
|
||||
}
|
||||
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
|
||||
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
|
||||
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
|
||||
|
||||
# Runtime: never silently downgrade a self-contained Server.
|
||||
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
|
||||
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
|
||||
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
|
||||
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
|
||||
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
|
||||
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
|
||||
}
|
||||
|
||||
# What the swap will carry over / drop (top-level files).
|
||||
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
|
||||
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
|
||||
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
|
||||
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
|
||||
}
|
||||
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
|
||||
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
|
||||
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
|
||||
|
||||
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
|
||||
if (-not $SkipFreeSpaceCheck) {
|
||||
$need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource)
|
||||
foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length }
|
||||
$need = [math]::Max($need * 2, $MinFreeGB * 1GB)
|
||||
$root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path)
|
||||
$free = (New-Object IO.DriveInfo $root).AvailableFreeSpace
|
||||
Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB))
|
||||
if ($free -lt $need) { Fail 'not enough free disk space for the backups' }
|
||||
}
|
||||
Say '== Preflight OK =='
|
||||
}
|
||||
|
||||
# --- stop / start ---------------------------------------------------------------------------------------------
|
||||
|
||||
function Wait-State([string]$Want, [int]$TimeoutSec) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
|
||||
$false
|
||||
}
|
||||
|
||||
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
|
||||
$deadline = (Get-Date).AddSeconds($TimeoutSec)
|
||||
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
|
||||
$false
|
||||
}
|
||||
|
||||
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
|
||||
function Stop-InstallProcesses($ctx, [string]$Why) {
|
||||
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
|
||||
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
|
||||
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
|
||||
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
|
||||
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
|
||||
$mine.Count
|
||||
}
|
||||
|
||||
function Stop-Gateway($ctx) {
|
||||
Say '-- Stop --'
|
||||
$state = Get-ServiceState
|
||||
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
|
||||
else {
|
||||
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
|
||||
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
|
||||
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
|
||||
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
|
||||
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
|
||||
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
|
||||
Warn 'still not STOPPED; killing the NSSM service host process'
|
||||
$svcPid = Get-ServiceHostPid
|
||||
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
|
||||
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
|
||||
}
|
||||
}
|
||||
if (-not (Wait-NoGatewayProcess $ctx 20)) {
|
||||
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
|
||||
[void](Stop-InstallProcesses $ctx 'left over after the stop')
|
||||
[void](Wait-NoGatewayProcess $ctx 10)
|
||||
}
|
||||
}
|
||||
}
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'after stop'
|
||||
Info 'stopped; no gateway/Worker process on the box'
|
||||
Assert-EnvUnchanged $ctx 'after the stop'
|
||||
}
|
||||
}
|
||||
|
||||
function Start-Gateway($ctx) {
|
||||
Say '-- Start --'
|
||||
if (-not $script:Dry) {
|
||||
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
|
||||
Assert-EnvUnchanged $ctx 'before the start'
|
||||
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
|
||||
Step "nssm start $ServiceName (once)" {
|
||||
# nssm exits non-zero with 'Unexpected status SERVICE_START_PENDING' when the app is slow to report RUNNING;
|
||||
# under ErrorActionPreference=Stop its stderr line throws. The RUNNING wait below is the real gate.
|
||||
try { $null = & $NssmPath start $ServiceName 2>&1 } catch { Say "[info] nssm start: $($_.Exception.Message) -- deciding by the RUNNING wait below" }
|
||||
}
|
||||
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
|
||||
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
|
||||
Assert-EnvUnchanged $ctx 'after the start'
|
||||
}
|
||||
}
|
||||
|
||||
# --- backup / swap -----------------------------------------------------------------------------------------------
|
||||
|
||||
function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } }
|
||||
|
||||
function Backup-State($ctx) {
|
||||
Say '-- Backup (service stopped) --'
|
||||
$suffix = Get-Suffix
|
||||
$root = Split-Path -Parent $ctx.ServerDir
|
||||
$dir = Join-Path $root "_deploy-backup-$suffix"
|
||||
$script:BackupDir = $dir
|
||||
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
|
||||
New-Item -ItemType Directory -Path $dir -Force | Out-Null
|
||||
Protect-Directory $dir
|
||||
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
|
||||
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
|
||||
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
|
||||
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
|
||||
}
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($f in (Get-StoreFiles $db)) {
|
||||
$dst = "$f.bak-$suffix"
|
||||
Step "copy store file $f -> $dst (live file is never moved)" {
|
||||
Copy-Item -LiteralPath $f -Destination $dst -Force
|
||||
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
|
||||
}
|
||||
}
|
||||
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
|
||||
}
|
||||
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
|
||||
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
|
||||
}
|
||||
}
|
||||
|
||||
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
|
||||
|
||||
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
|
||||
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
|
||||
$liveFiles = @(Get-Files $Live)
|
||||
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
|
||||
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
|
||||
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($f in $liveFiles) {
|
||||
$inSource = $srcNames -contains $f.Name
|
||||
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
|
||||
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
|
||||
if ($inSource) { continue }
|
||||
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
|
||||
}
|
||||
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
|
||||
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
|
||||
}
|
||||
|
||||
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
|
||||
$bak = "$Live.bak-$(Get-Suffix)"
|
||||
$plan = Get-CarryPlan $Live $Source $KeepSettings
|
||||
$hashes = @{}
|
||||
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
|
||||
$sddlOld = Get-DirSddl $Live
|
||||
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
|
||||
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
|
||||
foreach ($f in $plan.Carry) {
|
||||
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
|
||||
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
|
||||
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
|
||||
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
|
||||
}
|
||||
}
|
||||
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
|
||||
if (-not $script:Dry) {
|
||||
$sddlNew = Get-DirSddl $Live
|
||||
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
|
||||
}
|
||||
$bak
|
||||
}
|
||||
|
||||
# --- verification ----------------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
|
||||
Say '== Verify =='
|
||||
if ($script:Dry) {
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
|
||||
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
|
||||
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
|
||||
return
|
||||
}
|
||||
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
|
||||
while (-not $ready -and (Get-Date) -lt $deadline) {
|
||||
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
|
||||
if (-not $ready) { Start-Sleep -Seconds 3 }
|
||||
}
|
||||
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
|
||||
Info '/health/ready = 200'
|
||||
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
|
||||
if ($hz -ne 200) { Fail "/healthz = $hz" }
|
||||
Info '/healthz = 200'
|
||||
|
||||
$sha = Get-ProductSha $ctx.Application
|
||||
Info "deployed Server SHA (ProductVersion): $sha"
|
||||
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
|
||||
|
||||
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
|
||||
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
|
||||
Info "watching the service for $StableSeconds s (gateway #$pid0)"
|
||||
$until = (Get-Date).AddSeconds($StableSeconds)
|
||||
do {
|
||||
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
|
||||
Start-Sleep -Seconds 5
|
||||
} while ((Get-Date) -lt $until)
|
||||
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
|
||||
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
|
||||
Assert-EnvUnchanged $ctx 'after the stability window'
|
||||
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
|
||||
}
|
||||
|
||||
# --- rollback ------------------------------------------------------------------------------------------------------------
|
||||
|
||||
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
|
||||
function Get-BackupKey([string]$Name) {
|
||||
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
|
||||
$null
|
||||
}
|
||||
|
||||
function Find-Backup([string]$Live, [string]$Stamp) {
|
||||
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
|
||||
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
|
||||
if ($Stamp) {
|
||||
$key = ($Stamp -replace '[^0-9]', '')
|
||||
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
|
||||
}
|
||||
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
|
||||
}
|
||||
|
||||
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
|
||||
Say '== Rollback =='
|
||||
$sb = Find-Backup $ctx.ServerDir $Stamp
|
||||
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
|
||||
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
|
||||
$wb = Find-Backup $ctx.WorkerDir $key
|
||||
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
|
||||
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
|
||||
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
|
||||
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
|
||||
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
|
||||
if ($fpFile.Count) {
|
||||
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
|
||||
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
|
||||
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
|
||||
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
|
||||
}
|
||||
Stop-Gateway $ctx
|
||||
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
|
||||
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
|
||||
$failed = "$($pair[0]).failed-$($script:Stamp)"
|
||||
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
|
||||
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
|
||||
}
|
||||
if ($Stores) {
|
||||
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
|
||||
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
|
||||
foreach ($ext in '', '-wal', '-shm') {
|
||||
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
|
||||
if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } }
|
||||
elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) {
|
||||
# a stale wal/shm must not pair with the restored db: move it aside, never delete
|
||||
Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") }
|
||||
}
|
||||
elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" }
|
||||
}
|
||||
}
|
||||
} else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' }
|
||||
Start-Gateway $ctx
|
||||
}
|
||||
|
||||
# --- main -------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
function Invoke-Main {
|
||||
$exit = 0
|
||||
$ctx = $null
|
||||
try {
|
||||
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
|
||||
$ctx = Get-Context
|
||||
Invoke-Preflight $ctx
|
||||
|
||||
if (-not $script:Dry) {
|
||||
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
|
||||
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
|
||||
catch { Fail "another deploy run holds $lockPath" }
|
||||
}
|
||||
|
||||
if ($Rollback) {
|
||||
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
|
||||
Invoke-Verify $ctx
|
||||
} else {
|
||||
Stop-Gateway $ctx
|
||||
try { Backup-State $ctx }
|
||||
catch {
|
||||
$err = $_
|
||||
if (-not $script:Dry) {
|
||||
Warn 'backup failed before anything was swapped - starting the unchanged build again'
|
||||
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
|
||||
}
|
||||
throw $err
|
||||
}
|
||||
$touched = $false
|
||||
try {
|
||||
$touched = $true
|
||||
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
|
||||
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
|
||||
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
|
||||
Start-Gateway $ctx
|
||||
Invoke-Verify $ctx
|
||||
} catch {
|
||||
$err = $_
|
||||
Say "[FAIL] $(Hide $err.Exception.Message)"
|
||||
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
|
||||
Warn 'upgrade failed - rolling back automatically'
|
||||
$script:EnvWarnOnly = $true
|
||||
try {
|
||||
Invoke-RollbackSwap $ctx $script:Stamp $false $true
|
||||
Invoke-Verify $ctx $true
|
||||
Warn 'ROLLED BACK to the previous build; original failure follows'
|
||||
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
|
||||
throw $err
|
||||
}
|
||||
}
|
||||
Say '== DONE =='
|
||||
} catch {
|
||||
$exit = 1
|
||||
$m = Hide $_.Exception.Message
|
||||
$script:Notes.Add("ERROR: $m")
|
||||
Write-Host "ERROR: $m" -ForegroundColor Red
|
||||
} finally {
|
||||
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
|
||||
try {
|
||||
if (-not $script:Dry -and $ctx) {
|
||||
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
|
||||
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
|
||||
Write-Host "deploy notes: $notes"
|
||||
}
|
||||
} catch { }
|
||||
$script:Redact.Clear()
|
||||
}
|
||||
$exit
|
||||
}
|
||||
|
||||
# Dot-sourcing (the dry-run harness) only defines the functions.
|
||||
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }
|
||||
@@ -0,0 +1,117 @@
|
||||
{
|
||||
"Serilog": {
|
||||
"Using": [
|
||||
"Serilog.Sinks.Console",
|
||||
"Serilog.Sinks.File"
|
||||
],
|
||||
"MinimumLevel": {
|
||||
"Default": "Information",
|
||||
"Override": {
|
||||
"Microsoft.AspNetCore": "Warning"
|
||||
}
|
||||
},
|
||||
"WriteTo": [
|
||||
{
|
||||
"Name": "Console"
|
||||
},
|
||||
{
|
||||
"Name": "File",
|
||||
"Args": {
|
||||
"path": "logs/mxgateway-.log",
|
||||
"rollingInterval": "Day",
|
||||
"retainedFileCountLimit": 30,
|
||||
"fileSizeLimitBytes": 52428800,
|
||||
"rollOnFileSizeLimit": true
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"AllowedHosts": "*",
|
||||
"Secrets": {
|
||||
"MasterKey": {
|
||||
"Source": "Environment",
|
||||
"EnvVarName": "ZB_SECRETS_MASTER_KEY"
|
||||
},
|
||||
"RunMigrationsOnStartup": true,
|
||||
"ResolveCacheTtl": "00:00:30"
|
||||
},
|
||||
"MxGateway": {
|
||||
"Authentication": {
|
||||
"Mode": "ApiKey",
|
||||
"PepperSecretName": "MxGateway:ApiKeyPepper",
|
||||
"RunMigrationsOnStartup": true
|
||||
},
|
||||
"Ldap": {
|
||||
"Enabled": true,
|
||||
"Server": "localhost",
|
||||
"Port": 3893,
|
||||
"Transport": "None",
|
||||
"AllowInsecure": true,
|
||||
"SearchBase": "dc=zb,dc=local",
|
||||
"ServiceAccountDn": "cn=serviceaccount,dc=zb,dc=local",
|
||||
"ServiceAccountPassword": "${secret:ldap/mxgateway/bind}",
|
||||
"UserNameAttribute": "cn",
|
||||
"DisplayNameAttribute": "cn",
|
||||
"GroupAttribute": "memberOf"
|
||||
},
|
||||
"Worker": {
|
||||
"ExecutablePath": "src\\ZB.MOM.WW.MxGateway.Worker\\bin\\x86\\Release\\ZB.MOM.WW.MxGateway.Worker.exe",
|
||||
"RequiredArchitecture": "X86",
|
||||
"StartupTimeoutSeconds": 30,
|
||||
"ShutdownTimeoutSeconds": 10,
|
||||
"HeartbeatCheckIntervalSeconds": 5,
|
||||
"HeartbeatGraceSeconds": 15,
|
||||
"HeartbeatStuckCeilingSeconds": 75,
|
||||
"WorkerHeartbeatSendIntervalSeconds": 5,
|
||||
"WorkerHeartbeatGraceSeconds": 15,
|
||||
"WorkerHeartbeatStuckCeilingSeconds": 75,
|
||||
"EventChannelFullModeTimeoutMilliseconds": 5000,
|
||||
"MaxMessageBytes": 16842752
|
||||
},
|
||||
"Sessions": {
|
||||
"DefaultCommandTimeoutSeconds": 30,
|
||||
"MaxSessions": 64,
|
||||
"MaxPendingCommandsPerSession": 128,
|
||||
"DefaultLeaseSeconds": 1800,
|
||||
"LeaseSweepIntervalSeconds": 30,
|
||||
"DetachGraceSeconds": 30,
|
||||
"UnattachedSessionGraceSeconds": 0,
|
||||
"AllowMultipleEventSubscribers": false,
|
||||
"MaxEventSubscribersPerSession": 8
|
||||
},
|
||||
"Events": {
|
||||
"QueueCapacity": 10000,
|
||||
"BackpressurePolicy": "FailFast",
|
||||
"ReplayBufferCapacity": 1024,
|
||||
"ReplayRetentionSeconds": 300,
|
||||
"ReplayBufferMaxBytes": 67108864
|
||||
},
|
||||
"Dashboard": {
|
||||
"Enabled": true,
|
||||
"AllowAnonymousLocalhost": true,
|
||||
"SnapshotIntervalMilliseconds": 1000,
|
||||
"RecentFaultLimit": 100,
|
||||
"RecentSessionLimit": 200,
|
||||
"ShowTagValues": false
|
||||
},
|
||||
"Protocol": {
|
||||
"WorkerProtocolVersion": 1,
|
||||
"MaxGrpcMessageBytes": 16777216
|
||||
},
|
||||
"Galaxy": {
|
||||
"ConnectionString": "Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;",
|
||||
"CommandTimeoutSeconds": 60,
|
||||
"DashboardRefreshIntervalSeconds": 30,
|
||||
"PersistSnapshot": true
|
||||
},
|
||||
"Alarms": {
|
||||
"Enabled": true,
|
||||
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
|
||||
"DefaultArea": "",
|
||||
"ReconcileIntervalSeconds": 30,
|
||||
"PollIntervalMilliseconds": 500,
|
||||
"MaxAlarmsPerFetch": 1024,
|
||||
"AckClientName": "MxGateway"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
# PROD MxAccessGateway first install (plan Phase 4 + S3A). Run ON the box, only after the owner's go for Phase 4.
|
||||
# -Mode dry : verify the zip, the MXAccess gate and the env, expand into .\dry\, run Install-MxGatewayService -DryRun.
|
||||
# Creates nothing outside this stage folder.
|
||||
# -Mode real : expand into E:\ApiInstall\MxAccessGateway\{app,worker}, put appsettings.json.prod in place, create the
|
||||
# KEK (or take -KekFrom), install the NSSM service STOPPED (never -Start: seed the LDAP secret first, S3A.4).
|
||||
# -ValuesDir holds one <KEY>.txt per '@file' line in mxgw.env (pepper, Galaxy connection string). It is read and never
|
||||
# printed; the caller deletes it afterwards. Secret values never go on argv.
|
||||
param(
|
||||
[Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode,
|
||||
[string]$ValuesDir,
|
||||
[string]$KekFrom
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'
|
||||
$stage = $PSScriptRoot
|
||||
$zip = Join-Path $stage 'mxgw-e74918cc.zip'
|
||||
$zipSha = 'F0999B75BEE196375AEB8E3E277A0E948C54A747FD5251D336461C9665D847F2'
|
||||
$root = 'E:\ApiInstall\MxAccessGateway'
|
||||
$kekDir = 'C:\ProgramData\ZB\kek'
|
||||
$kek = "$kekDir\mxgateway.kek"
|
||||
$nssm = 'E:\psrem-stage\prod-common\nssm.exe'
|
||||
$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
function Fail($m) { throw "ABORT: $m" }
|
||||
|
||||
Say "install-mxgw mode=$Mode box=$env:COMPUTERNAME"
|
||||
# 1. MXAccess gate (plan Q3): the PAR boxes have none until WinPlatform is deployed from the prod Galaxy.
|
||||
if (-not (Test-Path -LiteralPath $lmx)) { Fail "no $lmx - MXAccess not installed on this box (plan Q3 gate)" }
|
||||
Say "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12))"
|
||||
# 2. artifact
|
||||
$h = (Get-FileHash $zip -Algorithm SHA256).Hash
|
||||
if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" }
|
||||
Say 'zip sha256 ok'
|
||||
if (-not (Test-Path $nssm)) { Fail "nssm not staged at $nssm" }
|
||||
# 3. env: placeholders + @file values
|
||||
$lines = Get-Content (Join-Path $stage 'mxgw.env')
|
||||
$left = @($lines | Where-Object { $_ -notmatch '^\s*#' -and $_ -match '__Q\d+_' })
|
||||
if ($left.Count) { Fail "unresolved placeholders in mxgw.env: $(($left | ForEach-Object { ($_ -split '=')[0] }) -join ', ')" }
|
||||
$out = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($l in $lines) {
|
||||
if ($l -match '^\s*(#|$)') { continue }
|
||||
$k, $v = $l -split '=', 2
|
||||
if ($v -eq '@file') {
|
||||
if (-not $ValuesDir) { Fail "$k needs -ValuesDir" }
|
||||
$f = Join-Path $ValuesDir "$k.txt"
|
||||
if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" }
|
||||
$v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n")
|
||||
if (-not $v) { Fail "empty value file for $k" }
|
||||
}
|
||||
$out.Add("$k=$v")
|
||||
}
|
||||
Say "env entries: $($out.Count)"
|
||||
if ($out | Where-Object { $_ -match '_(DEV|QA)=' }) { Fail 'a _DEV/_QA group name is in a PROD env (plan S3A.2)' }
|
||||
|
||||
# 4. layout
|
||||
if ($Mode -eq 'dry') { $base = Join-Path $stage 'dry' } else { $base = $root }
|
||||
$app = Join-Path $base 'app'; $worker = Join-Path $base 'worker'
|
||||
if ($Mode -eq 'real') {
|
||||
if ((Test-Path $app) -or (Test-Path $worker)) { Fail "$app or $worker already exists - this is the first-install script; use Upgrade-MxGatewayService.ps1" }
|
||||
if (Get-Service MxAccessGw -ErrorAction SilentlyContinue) { Fail 'service MxAccessGw already exists' }
|
||||
} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force }
|
||||
$tmp = Join-Path $stage "expand-$(Get-Date -Format yyyyMMddTHHmmss)"
|
||||
Expand-Archive -LiteralPath $zip -DestinationPath $tmp
|
||||
New-Item -ItemType Directory -Force $base | Out-Null
|
||||
Move-Item (Join-Path $tmp 'Server') $app
|
||||
Move-Item (Join-Path $tmp 'Worker') $worker
|
||||
Remove-Item $tmp -Recurse -Force
|
||||
Copy-Item (Join-Path $stage 'appsettings.json.prod') (Join-Path $app 'appsettings.json') -Force
|
||||
if (Select-String -Path (Join-Path $app 'appsettings.json') -Pattern 'GwAdmin|GwReader' -Quiet) { Fail 'GwAdmin/GwReader still in appsettings.json' }
|
||||
Say "expanded to $base (app: $(@(Get-ChildItem $app -Recurse -File).Count) files, worker: $(@(Get-ChildItem $worker -Recurse -File).Count) files)"
|
||||
foreach ($e in (Join-Path $app 'ZB.MOM.WW.MxGateway.Server.exe'), (Join-Path $worker 'MxGateway.Worker.exe')) {
|
||||
Say "$(Split-Path $e -Leaf) $((Get-Item $e).VersionInfo.ProductVersion)" }
|
||||
|
||||
# 5. KEK (32 raw random bytes; Administrators + SYSTEM only)
|
||||
if ($Mode -eq 'real') {
|
||||
New-Item -ItemType Directory -Force $kekDir | Out-Null
|
||||
& icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null
|
||||
if (Test-Path $kek) { Say "KEK exists, kept: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" }
|
||||
elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say "KEK copied from -KekFrom: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" }
|
||||
else {
|
||||
$b = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b)
|
||||
[IO.File]::WriteAllBytes($kek, $b); Say "KEK generated: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))"
|
||||
}
|
||||
if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' }
|
||||
}
|
||||
|
||||
# 6. service (stopped). Env goes through a protected temp file that is always deleted.
|
||||
$envTmp = Join-Path $stage "_env-$PID.tmp"
|
||||
try {
|
||||
[IO.File]::WriteAllLines($envTmp, $out)
|
||||
& icacls.exe $envTmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
|
||||
$a = @{ ServerDir = $app; WorkerDir = $worker; NssmPath = $nssm; DataDir = "$root\data"; LogDir = "$root\logs"
|
||||
EnvFile = $envTmp; WorkerExeName = 'MxGateway.Worker.exe' }
|
||||
if ($Mode -eq 'dry') { $a.DryRun = $true }
|
||||
& (Join-Path $stage 'Install-MxGatewayService.ps1') @a
|
||||
$rc = $LASTEXITCODE
|
||||
} finally { Remove-Item $envTmp -Force -ErrorAction SilentlyContinue }
|
||||
if ($rc -ne 0) { Fail "Install-MxGatewayService exited $rc" }
|
||||
if ($Mode -eq 'real') { Say 'INSTALLED STOPPED. Next: seed nam-ldap-svc-password (prod-common\seed-secret-prod.ps1), review the env, then start (plan S3A.4/S3A.5).' }
|
||||
Say "DONE mode=$Mode"
|
||||
@@ -0,0 +1,42 @@
|
||||
# MxAccessGateway NSSM env for PROD (all four boxes; QA vq03 key set + PROD plan §3A.3). KEY=VALUE, '#' comments.
|
||||
# @file = value read at install time from <ValuesDir>\<KEY>.txt (secrets; never in this file)
|
||||
# __Q2_..__ = open question Q2 (prod Galaxy Repository / alarm area); install-mxgw.ps1 refuses while any remain
|
||||
ASPNETCORE_ENVIRONMENT=Production
|
||||
DOTNET_ENVIRONMENT=Production
|
||||
Kestrel__Endpoints__Grpc__Url=http://0.0.0.0:5120
|
||||
Kestrel__Endpoints__Grpc__Protocols=Http2
|
||||
Kestrel__Endpoints__Dashboard__Url=http://0.0.0.0:5130
|
||||
Kestrel__Endpoints__Dashboard__Protocols=Http1
|
||||
MxGateway__Worker__ExecutablePath=E:\ApiInstall\MxAccessGateway\worker\MxGateway.Worker.exe
|
||||
MxGateway__Worker__RequiredArchitecture=X86
|
||||
MxGateway__Worker__WorkingDirectory=E:\ApiInstall\MxAccessGateway\worker
|
||||
MxGateway__Worker__MaxMessageBytes=16842752
|
||||
MxGateway__Protocol__MaxGrpcMessageBytes=16777216
|
||||
MxGateway__Galaxy__ConnectionString=@file
|
||||
MxGateway__Authentication__Mode=ApiKey
|
||||
MxGateway__Authentication__SqlitePath=E:\ApiInstall\MxAccessGateway\data\gateway-auth.db
|
||||
MxGateway__ApiKeyPepper=@file
|
||||
MxGateway__Alarms__Enabled=true
|
||||
MxGateway__Alarms__SubscriptionExpression=__Q2_ALARM_EXPRESSION__
|
||||
MxGateway__Ldap__Enabled=true
|
||||
MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com
|
||||
MxGateway__Ldap__Port=636
|
||||
MxGateway__Ldap__Transport=Ldaps
|
||||
MxGateway__Ldap__AllowInsecure=false
|
||||
MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636
|
||||
MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com
|
||||
MxGateway__Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com
|
||||
MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}
|
||||
MxGateway__Ldap__UserNameAttribute=sAMAccountName
|
||||
MxGateway__Ldap__DisplayNameAttribute=name
|
||||
MxGateway__Ldap__GroupAttribute=memberOf
|
||||
MxGateway__Ldap__BindMode=ServiceAccount
|
||||
MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_PROD=Administrator
|
||||
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_PROD=Viewer
|
||||
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_PROD=Viewer
|
||||
MxGateway__Dashboard__RequireHttpsCookie=false
|
||||
Secrets__SqlitePath=E:\ApiInstall\MxAccessGateway\data\secrets.db
|
||||
Secrets__MasterKey__Source=File
|
||||
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\mxgateway.kek
|
||||
MxGateway__Telemetry__Exporter=Otlp
|
||||
MxGateway__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
|
||||
@@ -0,0 +1,116 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54).
|
||||
|
||||
.DESCRIPTION
|
||||
Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\}
|
||||
Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit.
|
||||
ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit
|
||||
ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and
|
||||
Modify on data\ and logs\.
|
||||
Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service
|
||||
wrapper, so otelcol runs in console mode).
|
||||
Every listener in config.yaml binds 127.0.0.1. The script checks that after start.
|
||||
|
||||
.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt.
|
||||
.PARAMETER Tier DEV, QA or PROD (PROD added in the 2026-11-19 PROD stage copy).
|
||||
.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\.
|
||||
.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed.<ts> (kept).
|
||||
#>
|
||||
param(
|
||||
[Parameter(Mandatory)] [string]$StageDir,
|
||||
[ValidateSet('DEV','QA','PROD')] [string]$Tier,
|
||||
[string]$NssmPath,
|
||||
[switch]$Uninstall
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'
|
||||
$svcName = 'ZbOtelCollector'
|
||||
$root = 'E:\ApiInstall\OtelCollector'
|
||||
$acct = "NT SERVICE\$svcName"
|
||||
$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
|
||||
if ($Uninstall) {
|
||||
$s = Get-Service $svcName -ErrorAction SilentlyContinue
|
||||
if ($s) {
|
||||
if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) }
|
||||
& sc.exe delete $svcName | Out-Null
|
||||
Say "service $svcName deleted"
|
||||
}
|
||||
if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" }
|
||||
return
|
||||
}
|
||||
|
||||
if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' }
|
||||
if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" }
|
||||
if (Test-Path $root) { throw "$root already exists" }
|
||||
|
||||
# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git).
|
||||
$sums = @{}
|
||||
Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } |
|
||||
ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() }
|
||||
foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') {
|
||||
$h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash
|
||||
if ($h -ne $sums[$f]) { throw "hash mismatch $f" }
|
||||
Say "hash ok $f $($h.Substring(0,12))"
|
||||
}
|
||||
|
||||
# 2. Layout
|
||||
foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null }
|
||||
$tmp = Join-Path $StageDir 'x'
|
||||
if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force }
|
||||
Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp
|
||||
$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash
|
||||
if ($h -ne $exeSha) { throw "exe hash mismatch $h" }
|
||||
Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe')
|
||||
Remove-Item $tmp -Recurse -Force
|
||||
Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe')
|
||||
Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml')
|
||||
Say "layout ok, exe sha $($exeSha.Substring(0,12))"
|
||||
|
||||
# 3. Service
|
||||
$nssm = Join-Path $root 'bin\nssm.exe'
|
||||
& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null
|
||||
$set = @(
|
||||
@('AppParameters', "--config=$root\config\config.yaml"),
|
||||
@('AppDirectory', $root),
|
||||
@('DisplayName', 'ZB OpenTelemetry Collector'),
|
||||
@('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'),
|
||||
@('Start', 'SERVICE_AUTO_START'),
|
||||
@('AppStdout', "$root\logs\collector.log"),
|
||||
@('AppStderr', "$root\logs\collector.log"),
|
||||
@('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'),
|
||||
@('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000'))
|
||||
foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } }
|
||||
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1')
|
||||
& sc.exe config $svcName obj= $acct | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" }
|
||||
Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)"
|
||||
|
||||
# 4. ACL
|
||||
& icacls $root /inheritance:d | Out-Null
|
||||
& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null
|
||||
& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null
|
||||
foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null }
|
||||
(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() }
|
||||
|
||||
# 5. Start + checks
|
||||
Start-Service $svcName
|
||||
$ok = $false
|
||||
for ($i = 0; $i -lt 30; $i++) {
|
||||
Start-Sleep 2
|
||||
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {}
|
||||
}
|
||||
Say "service $((Get-Service $svcName).Status); health_check ok=$ok"
|
||||
if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' }
|
||||
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId
|
||||
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||
$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId
|
||||
$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||
$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' }
|
||||
if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" }
|
||||
$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces'
|
||||
Say "OTLP/HTTP empty POST -> $($r.StatusCode)"
|
||||
$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false
|
||||
Say "collector.log error lines: $(@($err).Count)"
|
||||
Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) }
|
||||
@@ -0,0 +1,65 @@
|
||||
<#
|
||||
.SYNOPSIS
|
||||
Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent.
|
||||
.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window.
|
||||
.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP).
|
||||
#>
|
||||
param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20)
|
||||
$ErrorActionPreference = 'Continue'
|
||||
$root = 'E:\ApiInstall\OtelCollector'
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'"
|
||||
Say "service $($s.State) $($s.StartMode) $($s.StartName)"
|
||||
try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" }
|
||||
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe'
|
||||
if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
|
||||
" collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) }
|
||||
Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime }
|
||||
" E: free GB " + [int]((Get-PSDrive E).Free/1GB)
|
||||
|
||||
$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000
|
||||
function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue }
|
||||
foreach ($sig in 'metrics','traces','logs') {
|
||||
$f = "$root\data\$sig.jsonl"
|
||||
if (-not (Test-Path $f)) { Say "${sig}: no file"; continue }
|
||||
$agg = @{}
|
||||
foreach ($line in (Get-Content $f -Tail $Tail)) {
|
||||
try { $o = $line | ConvertFrom-Json } catch { continue }
|
||||
$rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs }
|
||||
foreach ($r in $rs) {
|
||||
$key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role')
|
||||
if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } }
|
||||
$a = $agg[$key]
|
||||
if ($sig -eq 'metrics') {
|
||||
foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1
|
||||
foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } }
|
||||
} elseif ($sig -eq 'traces') {
|
||||
foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } }
|
||||
} else {
|
||||
foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } }
|
||||
}
|
||||
}
|
||||
}
|
||||
Say "$sig (last $Tail export requests):"
|
||||
foreach ($k in ($agg.Keys | Sort-Object)) {
|
||||
$a = $agg[$k]
|
||||
$newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' }
|
||||
$fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' }
|
||||
" $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh"
|
||||
if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') }
|
||||
}
|
||||
}
|
||||
try {
|
||||
$p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n"
|
||||
$jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name
|
||||
Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" }
|
||||
} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" }
|
||||
try {
|
||||
$q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n"
|
||||
Say "collector self-metrics:"
|
||||
$q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') }
|
||||
} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" }
|
||||
foreach ($port in $AppPorts) {
|
||||
$c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} }
|
||||
" app :$port /metrics $c"
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54).
|
||||
# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA),
|
||||
# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector.
|
||||
#
|
||||
# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317,
|
||||
# and nothing is reachable from the network. That is why the receivers have no auth and no TLS.
|
||||
# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md).
|
||||
#
|
||||
# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as
|
||||
# deployment.environment.name, so files copied off two tiers can still be told apart.
|
||||
#
|
||||
# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older
|
||||
# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB.
|
||||
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 127.0.0.1:13133
|
||||
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 127.0.0.1:4317
|
||||
http:
|
||||
endpoint: 127.0.0.1:4318
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 5s
|
||||
limit_mib: 256
|
||||
spike_limit_mib: 64
|
||||
resource/tier:
|
||||
attributes:
|
||||
- key: deployment.environment.name
|
||||
value: ${env:ZB_TIER}
|
||||
action: upsert
|
||||
batch: {}
|
||||
|
||||
exporters:
|
||||
file/metrics:
|
||||
path: E:/ApiInstall/OtelCollector/data/metrics.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
file/traces:
|
||||
path: E:/ApiInstall/OtelCollector/data/traces.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
file/logs:
|
||||
path: E:/ApiInstall/OtelCollector/data/logs.jsonl
|
||||
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
|
||||
prometheus:
|
||||
endpoint: 127.0.0.1:8889
|
||||
resource_constant_labels:
|
||||
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name]
|
||||
metric_expiration: 15m
|
||||
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
logs:
|
||||
level: info
|
||||
metrics:
|
||||
level: basic
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 127.0.0.1
|
||||
port: 8888
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/metrics, prometheus]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/traces]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, resource/tier, batch]
|
||||
exporters: [file/logs]
|
||||
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"ScadaBridge": {
|
||||
"Node": {
|
||||
"Role": "Central",
|
||||
"NodeName": "central-a",
|
||||
"NodeHostname": "10.220.153.205",
|
||||
"RemotingPort": 8081
|
||||
},
|
||||
"_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.",
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@10.220.153.205:8081",
|
||||
"akka.tcp://scadabridge@10.220.153.210:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
"MinNrOfMembers": 1,
|
||||
"BootstrapGuard": {
|
||||
"Enabled": true
|
||||
}
|
||||
},
|
||||
"Database": {
|
||||
"ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}"
|
||||
},
|
||||
"_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.",
|
||||
"Security": {
|
||||
"Ldap": {
|
||||
"Enabled": true,
|
||||
"Server": "uswap-vdc-nam1.nam.zimmer.com",
|
||||
"Port": 636,
|
||||
"Transport": "Ldaps",
|
||||
"AllowInsecure": false,
|
||||
"FallbackServers": [
|
||||
"uswap-vdc-nam2.nam.zimmer.com:636"
|
||||
],
|
||||
"SearchBase": "DC=nam,DC=zimmer,DC=com",
|
||||
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
|
||||
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
|
||||
"UserNameAttribute": "sAMAccountName",
|
||||
"DisplayNameAttribute": "name",
|
||||
"GroupAttribute": "memberOf",
|
||||
"BindMode": "ServiceAccount"
|
||||
},
|
||||
"JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}",
|
||||
"JwtExpiryMinutes": 25,
|
||||
"IdleTimeoutMinutes": 60,
|
||||
"RequireHttpsCookie": false,
|
||||
"ForwardedHeaders": {
|
||||
"Enabled": false
|
||||
}
|
||||
},
|
||||
"_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).",
|
||||
"Communication": {
|
||||
"DeploymentTimeout": "00:02:00",
|
||||
"LifecycleTimeout": "00:00:30",
|
||||
"QueryTimeout": "00:00:30",
|
||||
"TransportHeartbeatInterval": "00:00:05",
|
||||
"TransportFailureThreshold": "00:00:15",
|
||||
"CentralFetchBaseUrl": "http://10.220.153.205:8085"
|
||||
},
|
||||
"ManagementService": {
|
||||
"CommandTimeout": "00:00:30",
|
||||
"LongRunningCommandTimeout": "00:05:00"
|
||||
},
|
||||
"HealthMonitoring": {
|
||||
"ReportInterval": "00:00:30",
|
||||
"OfflineTimeout": "00:01:00"
|
||||
},
|
||||
"InboundApi": {
|
||||
"DefaultMethodTimeout": "00:00:30",
|
||||
"ApiKeyStore": {
|
||||
"Provider": "SqlServer"
|
||||
},
|
||||
"StandbyForwardPeerEndpoints": [
|
||||
"http://10.220.153.210:8085"
|
||||
]
|
||||
},
|
||||
"Notification": {
|
||||
"SmtpServer": "REPLACE_WITH_SMTP_HOST",
|
||||
"SmtpPort": 25,
|
||||
"AuthMode": "None",
|
||||
"FromAddress": "scada-notifications@zimmer.com"
|
||||
},
|
||||
"NotificationOutbox": {
|
||||
"DispatchInterval": "00:00:10",
|
||||
"DispatchBatchSize": 100,
|
||||
"StuckAgeThreshold": "00:10:00",
|
||||
"TerminalRetention": "365.00:00:00",
|
||||
"PurgeInterval": "1.00:00:00",
|
||||
"DeliveredKpiWindow": "00:01:00"
|
||||
},
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
"Secrets": {
|
||||
"Store": {
|
||||
"Provider": "SqlServer"
|
||||
}
|
||||
},
|
||||
"Serilog": {
|
||||
"WriteTo": [
|
||||
{},
|
||||
{
|
||||
"Args": {
|
||||
"path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"ScadaBridge": {
|
||||
"Node": {
|
||||
"Role": "Central",
|
||||
"NodeName": "central-b",
|
||||
"NodeHostname": "10.220.153.210",
|
||||
"RemotingPort": 8081
|
||||
},
|
||||
"_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.",
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@10.220.153.210:8081",
|
||||
"akka.tcp://scadabridge@10.220.153.205:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
"MinNrOfMembers": 1,
|
||||
"BootstrapGuard": {
|
||||
"Enabled": true
|
||||
}
|
||||
},
|
||||
"Database": {
|
||||
"ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}"
|
||||
},
|
||||
"_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.",
|
||||
"Security": {
|
||||
"Ldap": {
|
||||
"Enabled": true,
|
||||
"Server": "uswap-vdc-nam1.nam.zimmer.com",
|
||||
"Port": 636,
|
||||
"Transport": "Ldaps",
|
||||
"AllowInsecure": false,
|
||||
"FallbackServers": [
|
||||
"uswap-vdc-nam2.nam.zimmer.com:636"
|
||||
],
|
||||
"SearchBase": "DC=nam,DC=zimmer,DC=com",
|
||||
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
|
||||
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
|
||||
"UserNameAttribute": "sAMAccountName",
|
||||
"DisplayNameAttribute": "name",
|
||||
"GroupAttribute": "memberOf",
|
||||
"BindMode": "ServiceAccount"
|
||||
},
|
||||
"JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}",
|
||||
"JwtExpiryMinutes": 25,
|
||||
"IdleTimeoutMinutes": 60,
|
||||
"RequireHttpsCookie": false,
|
||||
"ForwardedHeaders": {
|
||||
"Enabled": false
|
||||
}
|
||||
},
|
||||
"_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).",
|
||||
"Communication": {
|
||||
"DeploymentTimeout": "00:02:00",
|
||||
"LifecycleTimeout": "00:00:30",
|
||||
"QueryTimeout": "00:00:30",
|
||||
"TransportHeartbeatInterval": "00:00:05",
|
||||
"TransportFailureThreshold": "00:00:15",
|
||||
"CentralFetchBaseUrl": "http://10.220.153.210:8085"
|
||||
},
|
||||
"ManagementService": {
|
||||
"CommandTimeout": "00:00:30",
|
||||
"LongRunningCommandTimeout": "00:05:00"
|
||||
},
|
||||
"HealthMonitoring": {
|
||||
"ReportInterval": "00:00:30",
|
||||
"OfflineTimeout": "00:01:00"
|
||||
},
|
||||
"InboundApi": {
|
||||
"DefaultMethodTimeout": "00:00:30",
|
||||
"ApiKeyStore": {
|
||||
"Provider": "SqlServer"
|
||||
},
|
||||
"StandbyForwardPeerEndpoints": [
|
||||
"http://10.220.153.205:8085"
|
||||
]
|
||||
},
|
||||
"Notification": {
|
||||
"SmtpServer": "REPLACE_WITH_SMTP_HOST",
|
||||
"SmtpPort": 25,
|
||||
"AuthMode": "None",
|
||||
"FromAddress": "scada-notifications@zimmer.com"
|
||||
},
|
||||
"NotificationOutbox": {
|
||||
"DispatchInterval": "00:00:10",
|
||||
"DispatchBatchSize": 100,
|
||||
"StuckAgeThreshold": "00:10:00",
|
||||
"TerminalRetention": "365.00:00:00",
|
||||
"PurgeInterval": "1.00:00:00",
|
||||
"DeliveredKpiWindow": "00:01:00"
|
||||
},
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
"Secrets": {
|
||||
"Store": {
|
||||
"Provider": "SqlServer"
|
||||
}
|
||||
},
|
||||
"Serilog": {
|
||||
"WriteTo": [
|
||||
{},
|
||||
{
|
||||
"Args": {
|
||||
"path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"ScadaBridge": {
|
||||
"Node": {
|
||||
"Role": "Site",
|
||||
"NodeName": "node-a",
|
||||
"NodeHostname": "10.201.75.86",
|
||||
"SiteId": "__Q8_SITE_ID__",
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083,
|
||||
"MetricsPort": 8084
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@10.201.75.86:8082",
|
||||
"akka.tcp://scadabridge@10.201.75.87:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
"MinNrOfMembers": 1,
|
||||
"BootstrapGuard": {
|
||||
"Enabled": true
|
||||
}
|
||||
},
|
||||
"_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.",
|
||||
"Database": {
|
||||
"SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
"ReconnectInterval": "00:00:05",
|
||||
"TagResolutionRetryInterval": "00:00:10",
|
||||
"WriteTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralGrpcEndpoints": [
|
||||
"http://10.220.153.205:8083",
|
||||
"http://10.220.153.210:8083"
|
||||
],
|
||||
"_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-<SiteId>.",
|
||||
"DeploymentTimeout": "00:02:00",
|
||||
"LifecycleTimeout": "00:00:30",
|
||||
"QueryTimeout": "00:00:30",
|
||||
"TransportHeartbeatInterval": "00:00:05",
|
||||
"TransportFailureThreshold": "00:00:15"
|
||||
},
|
||||
"HealthMonitoring": {
|
||||
"ReportInterval": "00:00:30",
|
||||
"OfflineTimeout": "00:01:00"
|
||||
},
|
||||
"SiteEventLog": {
|
||||
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db",
|
||||
"RetentionDays": 30,
|
||||
"MaxStorageMb": 1024,
|
||||
"PurgeScheduleCron": "0 2 * * *"
|
||||
},
|
||||
"Notification": {},
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
"Serilog": {
|
||||
"WriteTo": [
|
||||
{},
|
||||
{
|
||||
"Args": {
|
||||
"path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"LocalDb": {
|
||||
"Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db",
|
||||
"Replication": {
|
||||
"PeerAddress": "http://10.201.75.87:8083",
|
||||
"MaxBatchSize": 16,
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
},
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
{
|
||||
"ScadaBridge": {
|
||||
"Node": {
|
||||
"Role": "Site",
|
||||
"NodeName": "node-b",
|
||||
"NodeHostname": "10.201.75.87",
|
||||
"SiteId": "__Q8_SITE_ID__",
|
||||
"RemotingPort": 8082,
|
||||
"GrpcPort": 8083,
|
||||
"MetricsPort": 8084
|
||||
},
|
||||
"Cluster": {
|
||||
"SeedNodes": [
|
||||
"akka.tcp://scadabridge@10.201.75.87:8082",
|
||||
"akka.tcp://scadabridge@10.201.75.86:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
"MinNrOfMembers": 1,
|
||||
"BootstrapGuard": {
|
||||
"Enabled": true
|
||||
}
|
||||
},
|
||||
"_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.",
|
||||
"Database": {
|
||||
"SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db"
|
||||
},
|
||||
"DataConnection": {
|
||||
"ReconnectInterval": "00:00:05",
|
||||
"TagResolutionRetryInterval": "00:00:10",
|
||||
"WriteTimeout": "00:00:30"
|
||||
},
|
||||
"StoreAndForward": {
|
||||
"SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db"
|
||||
},
|
||||
"Communication": {
|
||||
"CentralGrpcEndpoints": [
|
||||
"http://10.220.153.205:8083",
|
||||
"http://10.220.153.210:8083"
|
||||
],
|
||||
"_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-<SiteId>.",
|
||||
"DeploymentTimeout": "00:02:00",
|
||||
"LifecycleTimeout": "00:00:30",
|
||||
"QueryTimeout": "00:00:30",
|
||||
"TransportHeartbeatInterval": "00:00:05",
|
||||
"TransportFailureThreshold": "00:00:15"
|
||||
},
|
||||
"HealthMonitoring": {
|
||||
"ReportInterval": "00:00:30",
|
||||
"OfflineTimeout": "00:01:00"
|
||||
},
|
||||
"SiteEventLog": {
|
||||
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db",
|
||||
"RetentionDays": 30,
|
||||
"MaxStorageMb": 1024,
|
||||
"PurgeScheduleCron": "0 2 * * *"
|
||||
},
|
||||
"Notification": {},
|
||||
"Logging": {
|
||||
"MinimumLevel": "Information"
|
||||
}
|
||||
},
|
||||
"Serilog": {
|
||||
"WriteTo": [
|
||||
{},
|
||||
{
|
||||
"Args": {
|
||||
"path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"LocalDb": {
|
||||
"Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db",
|
||||
"Replication": {
|
||||
"MaxBatchSize": 16,
|
||||
"MaxOplogRows": 250000,
|
||||
"MaxOplogAge": "2.00:00:00"
|
||||
}
|
||||
},
|
||||
"AuditLog": {
|
||||
"SiteWriter": {
|
||||
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
# PROD ScadaBridge first install of ONE node (plan Phase 5 = Central on vp03/vp04, Phase 6 = Site on par-vp03/04).
|
||||
# Run ON the box, only after the owner's go for that phase. Never starts the service.
|
||||
# -Mode dry : verify the zip, pick this box's config, check placeholders + values, expand into .\dry\, print the
|
||||
# service definition (env names only). Creates nothing outside this stage folder.
|
||||
# -Mode real : expand into E:\ApiInstall\ScadaBridge\{app,cli}, put this node's appsettings.<Role>.json in place,
|
||||
# create <role>\{data,logs}, the KEK (or -KekFrom; ONE scadabridge.kek for all four nodes), and the
|
||||
# ScadaBridge-<Role> service (LocalSystem, auto, env REG_MULTI_SZ, recovery 5s/5s/10s, failureflag 1).
|
||||
# -ValuesDir holds one <KEY>.txt per '@file' line of sb-<role>.env; never printed; the caller deletes it afterwards.
|
||||
param(
|
||||
[Parameter(Mandatory)][ValidateSet('Central','Site')][string]$Role,
|
||||
[Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode,
|
||||
[string]$ValuesDir,
|
||||
[string]$KekFrom
|
||||
)
|
||||
$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'
|
||||
$stage = $PSScriptRoot
|
||||
$zip = Join-Path $stage 'scadabridge-2e0ce81e.zip'
|
||||
$zipSha = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23'
|
||||
$root = 'E:\ApiInstall\ScadaBridge'
|
||||
$kekDir = 'C:\ProgramData\ZB\kek'
|
||||
$kek = "$kekDir\scadabridge.kek"
|
||||
$svc = "ScadaBridge-$Role"
|
||||
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
|
||||
function Fail($m) { throw "ABORT: $m" }
|
||||
|
||||
$boxMap = @{ 'WONDER-APP-VP03' = @('vp03','Central'); 'WONDER-APP-VP04' = @('vp04','Central')
|
||||
'WONDER-PAR-VP03' = @('parvp03','Site'); 'WONDER-PAR-VP04' = @('parvp04','Site') }
|
||||
$b = $boxMap[$env:COMPUTERNAME.ToUpperInvariant()]
|
||||
if (-not $b) { Fail "unknown box $env:COMPUTERNAME" }
|
||||
if ($b[1] -ne $Role) { Fail "$env:COMPUTERNAME is a $($b[1]) box in the PROD topology, not $Role" }
|
||||
Say "install-sb role=$Role mode=$Mode box=$env:COMPUTERNAME"
|
||||
|
||||
# 1. artifact
|
||||
$h = (Get-FileHash $zip -Algorithm SHA256).Hash
|
||||
if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" }
|
||||
Say 'zip sha256 ok'
|
||||
# 2. this node's config
|
||||
$cfg = Join-Path $stage "appsettings.$Role.json.prod-$($b[0])"
|
||||
$cfgText = [IO.File]::ReadAllText($cfg)
|
||||
if ($cfgText -match '__Q\d+_[A-Z_]+__') { Fail "unresolved placeholder $($Matches[0]) in $(Split-Path $cfg -Leaf)" }
|
||||
if ($cfgText -match '_(DEV|QA)"') { Fail 'a _DEV/_QA name in a PROD config' }
|
||||
$null = $cfgText | ConvertFrom-Json
|
||||
Say "config $(Split-Path $cfg -Leaf) ok"
|
||||
# 3. env
|
||||
$lines = Get-Content (Join-Path $stage "sb-$($Role.ToLowerInvariant()).env")
|
||||
$envList = New-Object System.Collections.Generic.List[string]
|
||||
foreach ($l in $lines) {
|
||||
if ($l -match '^\s*(#|$)') { continue }
|
||||
$k, $v = $l -split '=', 2
|
||||
if ($v -match '__Q\d+_') { Fail "unresolved placeholder in env $k" }
|
||||
if ($v -eq '@file') {
|
||||
if (-not $ValuesDir) { Fail "$k needs -ValuesDir" }
|
||||
$f = Join-Path $ValuesDir "$k.txt"
|
||||
if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" }
|
||||
$v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n")
|
||||
if (-not $v) { Fail "empty value file for $k" }
|
||||
if ($k -like '*JwtSigningKey' -and $v.Length -lt 32) { Fail 'JwtSigningKey shorter than 32 characters' }
|
||||
}
|
||||
$envList.Add("$k=$v")
|
||||
}
|
||||
Say "env entries: $($envList.Count) ($((($envList | ForEach-Object { ($_ -split '=')[0] }) -join ', ')))"
|
||||
|
||||
# 4. layout
|
||||
$base = if ($Mode -eq 'dry') { Join-Path $stage 'dry' } else { $root }
|
||||
$app = Join-Path $base 'app'; $cli = Join-Path $base 'cli'
|
||||
if ($Mode -eq 'real') {
|
||||
if (Test-Path $app) { Fail "$app already exists - first-install script only" }
|
||||
if (Get-Service $svc -ErrorAction SilentlyContinue) { Fail "service $svc already exists" }
|
||||
} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force }
|
||||
New-Item -ItemType Directory -Force $base | Out-Null
|
||||
Expand-Archive -LiteralPath $zip -DestinationPath $base
|
||||
Copy-Item $cfg (Join-Path $app "appsettings.$Role.json") -Force
|
||||
$roleDir = Join-Path $base $Role.ToLowerInvariant()
|
||||
New-Item -ItemType Directory -Force (Join-Path $roleDir 'data'), (Join-Path $roleDir 'logs') | Out-Null
|
||||
$exe = Join-Path $app 'ZB.MOM.WW.ScadaBridge.Host.exe'
|
||||
Say "expanded to ${base}: app $(@(Get-ChildItem $app -Recurse -File).Count) files, cli $(@(Get-ChildItem $cli -Recurse -File).Count) files; Host $((Get-Item $exe).VersionInfo.ProductVersion)"
|
||||
|
||||
# 5. KEK
|
||||
if ($Mode -eq 'real') {
|
||||
New-Item -ItemType Directory -Force $kekDir | Out-Null
|
||||
& icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null
|
||||
if (Test-Path $kek) { Say "KEK exists, kept" }
|
||||
elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say 'KEK copied from -KekFrom' }
|
||||
else {
|
||||
if ($b[0] -ne 'vp03') { Fail 'only vp03 generates scadabridge.kek; every other node takes the same file via -KekFrom' }
|
||||
$r = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($r); [IO.File]::WriteAllBytes($kek, $r); Say 'KEK generated'
|
||||
}
|
||||
if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' }
|
||||
Say "KEK sha256 $((Get-FileHash $kek).Hash.Substring(0,8)) (must match on all four nodes)"
|
||||
}
|
||||
|
||||
# 6. service, never started
|
||||
$disp = "ScadaBridge $Role Node"
|
||||
if ($Mode -eq 'dry') {
|
||||
Say "[DRYRUN] sc.exe create $svc binPath= `"$exe`" start= auto obj= LocalSystem DisplayName= `"$disp`""
|
||||
Say "[DRYRUN] HKLM\...\Services\$svc Environment = REG_MULTI_SZ, $($envList.Count) entries (values hidden)"
|
||||
Say "[DRYRUN] sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000; sc.exe failureflag $svc 1"
|
||||
} else {
|
||||
& sc.exe create $svc binPath= "`"$(Join-Path $root 'app\ZB.MOM.WW.ScadaBridge.Host.exe')`"" start= auto obj= LocalSystem DisplayName= $disp | Out-Null
|
||||
if ($LASTEXITCODE -ne 0) { Fail "sc create exited $LASTEXITCODE" }
|
||||
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc" -Name Environment -Type MultiString -Value $envList.ToArray()
|
||||
& sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
|
||||
& sc.exe failureflag $svc 1 | Out-Null
|
||||
$n = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc").Environment).Count
|
||||
if ($n -ne $envList.Count) { Fail "env read-back $n entries, expected $($envList.Count)" }
|
||||
Say "service $svc created STOPPED, env $n entries"
|
||||
if ($Role -eq 'Central') { Say 'Next: ConfigDb (sql\), role rows, zbsecrets schema, seed nam-ldap-svc-password ONCE, then start vp03, then vp04.' }
|
||||
else { Say 'Next: SB-GRPC-PSK-<siteId> in central store + Site entity; start BOTH site nodes together (plan Phase 6).' }
|
||||
}
|
||||
Say "DONE mode=$Mode"
|
||||
@@ -0,0 +1,15 @@
|
||||
# ScadaBridge-Central service env (REG_MULTI_SZ Environment) for PROD vp03 + vp04. QA key set minus MachineDataDb and
|
||||
# the LDAP password (§3A.3) and minus SitePsks (central reads SB-GRPC-PSK-<siteId> from its store, Phase 6).
|
||||
# @file = <ValuesDir>\<KEY>.txt at install time. JwtSigningKey and ApiKeyPepper MUST be the same on both centrals.
|
||||
SCADABRIDGE_CONFIG=Central
|
||||
ASPNETCORE_URLS=http://+:8085
|
||||
ScadaBridge__Database__ConfigurationDb=@file
|
||||
ScadaBridge__Security__JwtSigningKey=@file
|
||||
ScadaBridge__InboundApi__ApiKeyPepper=@file
|
||||
ScadaBridge__InboundApi__ApiKeyStore__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\inbound-api-keys.sqlite
|
||||
Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\secrets.db
|
||||
Secrets__MasterKey__Source=File
|
||||
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek
|
||||
Secrets__SqlServer__ConnectionString=@file
|
||||
ScadaBridge__Telemetry__Exporter=Otlp
|
||||
ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
|
||||
@@ -0,0 +1,11 @@
|
||||
# ScadaBridge-Site service env for PROD par-vp03 + par-vp04 (QA key set). Both values MUST be identical on both site
|
||||
# nodes; GrpcPsk also goes into central's store as SB-GRPC-PSK-<siteId>. Never reuse the PSK as the LocalDb key.
|
||||
# The KEK file must be the SAME scadabridge.kek as the centrals (deployed secrets are central-encrypted).
|
||||
SCADABRIDGE_CONFIG=Site
|
||||
ScadaBridge__Communication__GrpcPsk=@file
|
||||
LocalDb__Replication__ApiKey=@file
|
||||
Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\site\data\secrets.db
|
||||
Secrets__MasterKey__Source=File
|
||||
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek
|
||||
ScadaBridge__Telemetry__Exporter=Otlp
|
||||
ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
|
||||
@@ -0,0 +1,20 @@
|
||||
-- PROD ScadaBridge LDAP group -> role rows (plan §3A.3, D-P6). Run in the PROD ScadaBridge DB after the schema script
|
||||
-- (sqlcmd -I -b). Idempotent; refuses if a group is already mapped to a different role. Only _PROD names (§3A.2).
|
||||
-- Later changes go through the CLI (audited). Ids 1-5 are the migration's SCADA-* seed rows; they stay untouched.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
SET XACT_ABORT ON;
|
||||
BEGIN TRAN;
|
||||
DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL);
|
||||
INSERT @t VALUES
|
||||
(N'UG_WW_APP_ADMIN_PROD', N'Administrator'),
|
||||
(N'UG_WW_APP_IDE_PROD', N'Designer'),
|
||||
(N'UG_WW_APP_IDE_RO_PROD', N'Viewer'),
|
||||
(N'UG_WW_BTMGR_RO_PROD', N'Viewer'),
|
||||
(N'UG_WW_HIST_ADMIN_PROD', N'Deployer'); -- interim until UG_WW_APP_DEPLOY_PROD exists (D-P6)
|
||||
IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role)
|
||||
THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1;
|
||||
INSERT dbo.LdapGroupMappings (LdapGroupName, Role)
|
||||
SELECT t.LdapGroupName, t.Role FROM @t t
|
||||
WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName);
|
||||
COMMIT;
|
||||
SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id;
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,44 @@
|
||||
-- ScadaBridge Central shared secret store (Secrets:Store:Provider=SqlServer), schema "zbsecrets", version 1.
|
||||
-- Verbatim DDL from ZB.MOM.WW.Secrets.Replicator.SqlServer SqlServerSecretsSchema.CreateSchemaDdl (unchanged since 0.2.0,
|
||||
-- so identical in the 0.6.2 package ScadaBridge 2e0ce81e pins), plus the version-row MERGE its migrator writes.
|
||||
-- WHY: the app creates this at first start, but nam-ldap-svc-password must be seeded BEFORE the first start (§3A.4) and the
|
||||
-- headless CLI runs no migrator. Idempotent; run in the ScadaBridge DB after scadabridge-schema-2e0ce81e.sql.
|
||||
SET QUOTED_IDENTIFIER ON;
|
||||
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'zbsecrets')
|
||||
EXEC(N'CREATE SCHEMA [zbsecrets]');
|
||||
GO
|
||||
IF OBJECT_ID(N'[zbsecrets].[schema_version]', N'U') IS NULL
|
||||
CREATE TABLE [zbsecrets].[schema_version] (
|
||||
id INT NOT NULL PRIMARY KEY CHECK (id = 1),
|
||||
version INT NOT NULL,
|
||||
applied_utc NVARCHAR(33) NOT NULL
|
||||
);
|
||||
IF OBJECT_ID(N'[zbsecrets].[secret]', N'U') IS NULL
|
||||
CREATE TABLE [zbsecrets].[secret] (
|
||||
name NVARCHAR(450) NOT NULL PRIMARY KEY,
|
||||
description NVARCHAR(MAX) NULL,
|
||||
content_type NVARCHAR(64) NOT NULL,
|
||||
ciphertext VARBINARY(MAX) NOT NULL,
|
||||
nonce VARBINARY(MAX) NOT NULL,
|
||||
tag VARBINARY(MAX) NOT NULL,
|
||||
wrapped_dek VARBINARY(MAX) NOT NULL,
|
||||
wrap_nonce VARBINARY(MAX) NOT NULL,
|
||||
wrap_tag VARBINARY(MAX) NOT NULL,
|
||||
kek_id NVARCHAR(256) NOT NULL,
|
||||
revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0,
|
||||
is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0,
|
||||
deleted_utc NVARCHAR(33) NULL,
|
||||
created_utc NVARCHAR(33) NOT NULL,
|
||||
updated_utc NVARCHAR(33) NOT NULL,
|
||||
created_by NVARCHAR(256) NULL,
|
||||
updated_by NVARCHAR(256) NULL
|
||||
);
|
||||
IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = N'ix_secret_updated_utc' AND object_id = OBJECT_ID(N'[zbsecrets].[secret]'))
|
||||
CREATE INDEX ix_secret_updated_utc ON [zbsecrets].[secret] (updated_utc);
|
||||
MERGE [zbsecrets].[schema_version] AS target
|
||||
USING (SELECT 1 AS id, 1 AS version, CONVERT(nvarchar(33), FORMAT(SYSUTCDATETIME(), 'yyyy-MM-ddTHH:mm:ss.fffffff') + N'+00:00') AS applied_utc) AS source
|
||||
ON target.id = source.id
|
||||
WHEN NOT MATCHED THEN INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
|
||||
GO
|
||||
SELECT s.name AS [schema], t.name AS [table] FROM sys.tables t JOIN sys.schemas s ON s.schema_id = t.schema_id WHERE s.name = N'zbsecrets';
|
||||
SELECT id, version, applied_utc FROM [zbsecrets].[schema_version];
|
||||
Reference in New Issue
Block a user