deploy(prod): stage the PROD install pack on app-vp03/04 + par-vp03/04 over SMB (nothing executed)

Configs, env templates, install wrappers (dry/real, never start), SQL (65-migration schema, zbsecrets v1, _PROD role rows),
preflight, seed script, patched mxgw Install script (worker name, file KEK, connstr mask). Plan: Phase 3 gate corrected.

Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
Joseph Doherty
2026-11-19 17:26:12 -05:00
parent 7261b18539
commit bfc369062c
22 changed files with 4539 additions and 3 deletions
@@ -0,0 +1,95 @@
# 2026-11-19: PROD install pack staged on wonder-app-vp03/vp04 + wonder-par-vp03/vp04 (SMB only)
**Owner request (2026-11-19):** "prep the application files, scripts, configs and copy them to the 4 servers for prod".
Plan: [`docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md`](../../plans/2026-11-19-prod-scadabridge-mxgw-install.md).
**What was done:** files copied over SMB (445) as `NAM\adm_dohertj2`. **Nothing ran on any PROD box**: no WinRM, no
service, no install, no KEK, no secret. The PROD deploy hold and every phase gate in the plan still apply. Each script
below is run on the box only after the owner's go for its phase.
## Artifacts (the QA-soaked builds, byte-identical; plan D-P2)
| File | sha256 | Source |
|---|---|---|
| `scadabridge-2e0ce81e.zip` (app + cli, self-contained win-x64) | `6c706b0a…6572e23` | the zip QA vq03/vq04 run since 2026-11-19 |
| `mxgw-e74918cc.zip` (Server + x86 Worker `MxGateway.Worker.exe`, `0.1.2+e74918cc`) | `f0999b75…5d847f2` | the zip DEV vd03 + QA run |
| `otelcol-contrib_0.162.0_windows_amd64.zip` | `97a675ee…e496a167` | QA vq03 `E:\psrem-stage\otel-54` (scadaproj#54) |
| Secrets CLI 0.6.6 `ZB.MOM.WW.Secrets.Cli.exe` | `36dbd512…` | = QA `E:\ApiInstall\_tools\secret-0.6.6` |
| Secrets CLI 0.6.6 + SqlServer provider (centrals only) | `421f95e9…` | = QA `…\secret-0.6.6-sql` (QA finding 2) |
| `nssm.exe` | `eee9c44c…` | = QA `C:\ProgramData\ZB\qa-build\tools\nssm.exe` |
Every staged folder carries a `SHA256SUMS.txt`. Remote sizes were checked file by file over SMB (150/150), and the two
app zips plus the CLI were pulled back from vp04 and par-vp04 and re-hashed.
## Layout on each box
| Path | vp03 / vp04 | par-vp03 / par-vp04 |
|---|---|---|
| `E:\psrem-stage\prod-common\` | `nssm.exe`, `preflight-prod.ps1`, `seed-secret-prod.ps1` | same |
| `E:\psrem-stage\mxgw-e74918cc\` | zip, `appsettings.json.prod`, `mxgw.env`, `install-mxgw.ps1`, `Install-`/`Upgrade-MxGatewayService.ps1` | same (install refuses until MXAccess exists, Q3) |
| `E:\psrem-stage\sb-2e0ce81e\` | zip, `appsettings.Central.json.prod-<box>`, `sb-central.env`, `install-sb.ps1`, `sql\` | zip, `appsettings.Site.json.prod-<box>`, `sb-site.env`, `install-sb.ps1` |
| `E:\psrem-stage\otel-54\` | collector zip, `config.yaml`, `Install-OtelCollector.ps1` (PROD tier added), `Test-OtelCollector.ps1` | same |
| `E:\ApiInstall\_tools\` | `secret-0.6.6\`, `secret-0.6.6-sql\` | `secret-0.6.6\` |
`E:\ApiInstall` now exists only for `_tools`; there is still no `app` folder and no service, so both install scripts see a
greenfield box.
## Configs (sources in this folder)
- **mxgw** `mxgw.env`: QA vq03's key set plus the §3A.3 LDAP block. Differences: `ASPNETCORE_ENVIRONMENT=Production` (login
hard-stops on), `_PROD` `GroupToRole`, and the file KEK `C:\ProgramData\ZB\kek\mxgateway.kek`. `appsettings.json.prod` = the
zip's `Server\appsettings.json` minus `Dashboard:GroupToRole` (`GwAdmin`/`GwReader`).
- **ScadaBridge centrals**: per node `appsettings.Central.json.prod-vp03|vp04` (QA shape, IPv4 literals, self-first seeds,
`leader-auto-down`, BootstrapGuard on, NAM LDAPS block, `RequireHttpsCookie=false`, shared SQL secret + API-key stores,
CentralGrpcPort left at the default **8083**: QA's 8086 only existed because of co-location). `sb-central.env` is QA's key
set minus `MachineDataDb` (gone since 3808b49f), minus the LDAP password env (§3A.3), and minus `SitePsks`
(central reads `SB-GRPC-PSK-<siteId>` from its store, Phase 6).
- **ScadaBridge sites**: `appsettings.Site.json.prod-parvp03|parvp04` (node-a/node-b, `PeerAddress` on node-a only, both
centrals at `:8083`, absolute data paths). `sb-site.env` = QA's site key set.
- **SQL (centrals, `sql\`)**: `scadabridge-schema-2e0ce81e.sql` (full, non-idempotent, 65 migrations, from `dotnet ef
migrations script 0 20261115221558_RemoveAuditLogHashChain` at 2e0ce81e; run with `sqlcmd -I -b`),
`zbsecrets-schema-v1.sql`, `sb-role-mappings-prod.sql` (the five `_PROD` rows, D-P6).
### Placeholders still open (the install scripts refuse while any remains)
| Placeholder / value | Where | Question |
|---|---|---|
| `__Q2_ALARM_EXPRESSION__` | `mxgw.env` | Q2 alarm area (`\\<GR node>\Galaxy!<area>`) |
| `MxGateway__Galaxy__ConnectionString` (value file) | install time | Q2 prod GR SQL |
| `__Q8_SITE_ID__` | both `appsettings.Site.json.prod-*` | Q8 SiteId |
| `ScadaBridge__Database__ConfigurationDb`, `Secrets__SqlServer__ConnectionString` (value files) | install time | Q1 prod SQL + `scadabridge_svc` |
| JWT key, inbound pepper (same on both centrals), mxgw pepper (shared by the PAR pair, pattern a), site PSK + LocalDb key (same on both sites) | install time, value files | generated then, stored in the Keychain |
When a placeholder is resolved, edit the file here, commit, and re-push that one file (its `SHA256SUMS.txt` line too).
## Scripts (dry first, always)
| Script | Does | Never does |
|---|---|---|
| `prod-common\preflight-prod.ps1 [-SqlHost] [-GrHost]` | read-only Phase 2 gates: disk, `LmxProxy.dll` + COM registration, NAM 636/389 + rootca + TLS, greenfield paths and ports, DNS, TCP path probes per §4 (open / refused / timeout), staged hashes | write anything |
| `mxgw-e74918cc\install-mxgw.ps1 -Mode dry\|real [-ValuesDir] [-KekFrom]` | MXAccess gate, zip hash, placeholder + `_DEV/_QA` refusal, expand to `app`/`worker`, NAM base config, 32-byte KEK, NSSM service via the patched `Install-MxGatewayService.ps1` | start the service |
| `sb-2e0ce81e\install-sb.ps1 -Role Central\|Site -Mode dry\|real [-ValuesDir] [-KekFrom]` | box/role check, zip hash, this node's config, env from value files, expand `app`/`cli`, `<role>\{data,logs}`, KEK (generated on vp03 only; every other node takes the same file), `sc.exe` service with recovery 5/5/10 s + failureflag | start the service |
| `prod-common\seed-secret-prod.ps1 -Target MxAccessGw\|ScadaBridge-Central -ValueFile` | seeds `nam-ldap-svc-password` into a stopped service's store (central: once, vp03, shared SQL store), reads back, deletes the value file | print the value |
`Install-MxGatewayService.ps1` here is mxgw `e74918cc` plus three marked patches: `-WorkerExeName` (the wonder Worker is
`MxGateway.Worker.exe`), File-KEK acceptance, and masking of connection strings in its env printout (the stock script
prints `MxGateway__Galaxy__ConnectionString` in clear). `Upgrade-MxGatewayService.ps1` is the `START_PENDING`-patched copy QA
used. Both repo fixes are still owed (mxaccessgw).
Checked on the Mac (pwsh 7, stubs for `nssm`/`icacls`): all eight scripts parse; `install-sb.ps1` dry runs pass for a
central and a site and refuse a wrong role and an open `__Q8_SITE_ID__`; `install-mxgw.ps1` dry run refuses an open `__Q2__`
and then runs `Install-MxGatewayService -DryRun` (39 env entries). That is not a Windows run; the on-box dry run is the gate.
## Order once the owner says go (plan phases)
1. Phase 2, each box: `preflight-prod.ps1` (needs the Q6 go for remote exec). Optional: `otel-54\Install-OtelCollector.ps1 -StageDir … -Tier PROD -NssmPath E:\psrem-stage\prod-common\nssm.exe`.
2. Phase 3 (Q1 answered): `sql\scadabridge-schema-2e0ce81e.sql`, `sql\zbsecrets-schema-v1.sql`, `sql\sb-role-mappings-prod.sql`; backup.
3. Phase 4, vp03/vp04 (par later, Q3): `install-mxgw.ps1 -Mode dry`, then `real`; `seed-secret-prod.ps1 -Target MxAccessGw`; start; §3A.5.
4. Phase 5: `install-sb.ps1 -Role Central` on vp03 (KEK generated), vp04 (`-KekFrom` the vp03 file); `seed-secret-prod.ps1 -Target ScadaBridge-Central` on vp03; start vp03, then vp04; §3A.5.
5. Phase 6: `install-sb.ps1 -Role Site` on both PAR boxes (`-KekFrom`, same PSK and LocalDb key); `SB-GRPC-PSK-<siteId>` into central; start both together.
## Cleanup (if PROD is abandoned)
`winrm smb rm -r <box> 'E$\psrem-stage\prod-common' 'E$\psrem-stage\mxgw-e74918cc' 'E$\psrem-stage\sb-2e0ce81e' 'E$\psrem-stage\otel-54' 'E$\ApiInstall\_tools'`
on each box. The push also created the parent folders `E:\psrem-stage` and `E:\ApiInstall`; the earlier 2026-11-19
SMB survey listed neither. Nothing else was created.
@@ -0,0 +1,65 @@
# PROD box preflight (plan Phase 2 gates + S3A LDAP gate). READ-ONLY: writes nothing, starts nothing, prints no secret.
# Run on each of the four boxes after the owner's go for remote exec (Q6). Optional -SqlHost/-GrHost for Q1/Q2 reachability.
param([string]$SqlHost, [string]$GrHost)
$ErrorActionPreference = 'Continue'; $ProgressPreference = 'SilentlyContinue'
$box = $env:COMPUTERNAME.ToUpperInvariant()
"== preflight-prod $box $(Get-Date -Format s)"
$ip = @{ 'WONDER-APP-VP03' = '10.220.153.205'; 'WONDER-APP-VP04' = '10.220.153.210'; 'WONDER-PAR-VP03' = '10.201.75.86'; 'WONDER-PAR-VP04' = '10.201.75.87' }
$central = $box -like 'WONDER-APP-*'
"role (planned): $(if ($central) { 'ScadaBridge Central + mxgw' } else { 'ScadaBridge Site + mxgw' })"
Get-CimInstance Win32_LogicalDisk -Filter "DriveType=3" | ForEach-Object { "disk $($_.DeviceID) free $([math]::Round($_.FreeSpace/1GB,1)) GB of $([math]::Round($_.Size/1GB,1)) GB" }
"os $((Get-CimInstance Win32_OperatingSystem).Caption) ps $($PSVersionTable.PSVersion)"
# MXAccess (Q3 gate for mxgw)
$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll'
if (Test-Path $lmx) { "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12)) (want 6000.0045.2657.1 / BD75A2FDE222)" } else { 'LmxProxy.dll ABSENT - mxgw cannot run here yet (Q3)' }
$clsid = Get-ItemProperty 'Registry::HKEY_CLASSES_ROOT\WOW6432Node\CLSID\*\ProgID' -ErrorAction SilentlyContinue | Where-Object { $_.'(default)' -like 'LMXProxy.LMXProxyServer*' }
"LMXProxy.LMXProxyServer COM registration: $(if ($clsid) { 'present' } else { 'absent' })"
# NAM LDAPS (S3A gate)
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') { foreach ($p in 636,389) {
$c = New-Object Net.Sockets.TcpClient; $ok = $c.ConnectAsync($h,$p).Wait(4000); $c.Dispose(); "reach $h`:$p $ok" } }
$root = @(Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like '*zimmer-rootca*' })
"zimmer-rootca in LocalMachine\Root: $($root.Count) $(($root | ForEach-Object { $_.Thumbprint + ' exp ' + $_.NotAfter.ToString('yyyy-MM-dd') }) -join '; ')"
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') {
try { $t = New-Object Net.Sockets.TcpClient($h,636); $script:err = $null
$s = New-Object Net.Security.SslStream($t.GetStream(), $false, { param($a,$b,$c,$e) $script:err = $e; $true })
$s.AuthenticateAsClient($h); "tls $h policyErrors=$script:err proto=$($s.SslProtocol)"; $s.Dispose(); $t.Dispose() } catch { "tls $h FAILED $($_.Exception.Message)" } }
# Local state: greenfield expected
foreach ($d in 'E:\ApiInstall','E:\ApiInstall\MxAccessGateway','E:\ApiInstall\ScadaBridge','C:\ProgramData\ZB\kek') { "path $d exists=$(Test-Path $d)" }
Get-Service | Where-Object { $_.Name -match 'MxAccess|MxGateway|ScadaBridge|ZbOtel|OtOpcUa|Historian' } | ForEach-Object { "service $($_.Name) $($_.Status)" }
$listen = Get-NetTCPConnection -State Listen -ErrorAction SilentlyContinue | Where-Object { $_.LocalPort -in 4317,5120,5130,8081,8082,8083,8084,8085,8889 }
foreach ($l in $listen) { "PORT IN USE $($l.LocalAddress):$($l.LocalPort) pid $($l.OwningProcess) $((Get-Process -Id $l.OwningProcess -ErrorAction SilentlyContinue).ProcessName)" }
if (-not $listen) { 'ports 4317 5120 5130 8081-8085 8889: all free' }
$fw = Get-NetFirewallProfile -ErrorAction SilentlyContinue | ForEach-Object { "$($_.Name)=$($_.Enabled)" }
"windows firewall profiles: $($fw -join ' ')"
# Name resolution + paths to the other boxes (plan S4). open = listener; refused = path open, nothing listening yet;
# timeout = probably filtered. Nothing listens yet on a greenfield box, so 'refused' is the good answer here.
function Probe($h, $p) {
$c = New-Object Net.Sockets.TcpClient
try { $ok = $c.ConnectAsync($h, $p).Wait(3000); if ($ok) { 'open' } else { 'timeout' } }
catch { $m = $_.Exception.InnerException; while ($m.InnerException) { $m = $m.InnerException }; if ($m.Message -match 'refused|actively') { 'refused' } else { "error: $($m.Message)" } }
finally { $c.Dispose() } }
foreach ($n in 'wonder-app-vp03','wonder-app-vp04','wonder-par-vp03','wonder-par-vp04') {
$f = "$n.zmr.zimmer.com"; try { $a = ([Net.Dns]::GetHostAddresses($f) | Select-Object -First 1).IPAddressToString } catch { $a = 'NXDOMAIN' }
"dns $f -> $a (plan $($ip[$n.ToUpperInvariant()]))" }
$targets = if ($central) {
@(@{h = $ip[($(if ($box -eq 'WONDER-APP-VP03') {'WONDER-APP-VP04'} else {'WONDER-APP-VP03'}))]; p = 8081, 8085, 8083},
@{h = '10.201.75.86'; p = 8083, 5120}, @{h = '10.201.75.87'; p = 8083, 5120})
} else {
@(@{h = $ip[($(if ($box -eq 'WONDER-PAR-VP03') {'WONDER-PAR-VP04'} else {'WONDER-PAR-VP03'}))]; p = 8082, 8083, 5120},
@{h = '10.220.153.205'; p = 8083, 5120}, @{h = '10.220.153.210'; p = 8083, 5120})
}
foreach ($t in $targets) { foreach ($p in $t.p) { "path -> $($t.h):$p $(Probe $t.h $p)" } }
if ($SqlHost) { "path -> SQL $SqlHost`:1433 $(Probe $SqlHost 1433)" }
if ($GrHost) { "path -> GR $GrHost`:1433 $(Probe $GrHost 1433)" }
# Staged artifacts
foreach ($sums in Get-ChildItem 'E:\psrem-stage' -Recurse -Filter 'SHA256SUMS.txt' -ErrorAction SilentlyContinue) {
foreach ($line in Get-Content $sums.FullName) { if ($line -match '^([0-9a-fA-F]{64})\s+\*?(.+)$') {
$f = Join-Path $sums.DirectoryName $Matches[2]
if (-not (Test-Path $f)) { "stage MISSING $f" } elseif ((Get-FileHash $f).Hash -ne $Matches[1].ToUpperInvariant()) { "stage HASH MISMATCH $f" } else { "stage ok $f" } } } }
foreach ($e in 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe','E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe') { if (Test-Path $e) { "tool $e sha256 $((Get-FileHash $e).Hash.Substring(0,12))" } }
"PREFLIGHT DONE"
@@ -0,0 +1,56 @@
# Seed nam-ldap-svc-password into this PROD box's stores BEFORE each service's first start (plan S3A.4). QA seed-secret.ps1
# retargeted. Each store's location and KEK come from the installed (stopped) service's own env, never assumed.
# -Target MxAccessGw : this box's mxgw SQLite store (all four boxes)
# -Target ScadaBridge-Central : the SHARED SQL Server store (zbsecrets), ONCE, on vp03 only, with the SQL-aware CLI
# (the headless 0.6.6 CLI writes SQLite only). Needs sql\zbsecrets-schema-v1.sql applied.
# The value arrives as -ValueFile over SMB (Keychain -> 0600 temp -> SMB), is ACL'd, read back, compared, then DELETED.
# It is never printed and never on argv.
param(
[Parameter(Mandatory)][ValidateSet('MxAccessGw','ScadaBridge-Central')][string]$Target,
[Parameter(Mandatory)][string]$ValueFile
)
$ErrorActionPreference = 'Stop'
$cli = 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe'
$cliSql = 'E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe'
$name = 'nam-ldap-svc-password'
$desc = 'NAM LDAP lookup account (svc_igntadmin, PROD interim D-P5)'
& icacls.exe $ValueFile /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
function SvcEnv($svc) {
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"
if (-not (Test-Path $k)) { throw "service $svc is not installed" }
$l = @((Get-ItemProperty $k -ErrorAction SilentlyContinue).Environment) + @((Get-ItemProperty "$k\Parameters" -ErrorAction SilentlyContinue).AppEnvironmentExtra) | Where-Object { $_ }
$h = @{}; foreach ($e in $l) { $i = $e.IndexOf('='); if ($i -gt 0) { $h[$e.Substring(0,$i)] = $e.Substring($i+1) } }; $h
}
$names = 'Secrets__SqlitePath','Secrets__MasterKey__Source','Secrets__MasterKey__FilePath','Secrets__MasterKey__EnvVarName','Secrets__Store__Provider','Secrets__SqlServer__ConnectionString','ZB_SECRETS_MASTER_KEY'
$fail = 0
try {
if ($Target -eq 'ScadaBridge-Central' -and $env:COMPUTERNAME -ne 'WONDER-APP-VP03') { throw 'the shared central store is seeded once, from vp03' }
$svc = Get-Service $Target; if ($svc.Status -ne 'Stopped') { throw "$Target is $($svc.Status); seed before the first start" }
$want = [IO.File]::ReadAllText($ValueFile).TrimEnd("`r", "`n")
if (-not $want) { throw 'value file is empty' }
"value file read, length $($want.Length)"
$e = SvcEnv $Target
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue; if ($e.ContainsKey($n)) { Set-Item "Env:$n" $e[$n] } }
$exe = $cli
if ($Target -eq 'ScadaBridge-Central') {
$env:Secrets__Store__Provider = 'SqlServer'; $exe = $cliSql
if (-not $env:Secrets__SqlServer__ConnectionString) { throw 'Central env has no Secrets__SqlServer__ConnectionString' }
}
if ($env:Secrets__MasterKey__Source -ne 'File') { throw "master key source '$($env:Secrets__MasterKey__Source)' is not File" }
if (-not (Test-Path $env:Secrets__MasterKey__FilePath)) { throw "KEK $($env:Secrets__MasterKey__FilePath) missing" }
$where = if ($Target -eq 'ScadaBridge-Central') { 'SqlServer shared store' } else { $env:Secrets__SqlitePath }
$tmp = "$ValueFile.nl"; [IO.File]::WriteAllText($tmp, $want)
& icacls.exe $tmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
$out = & cmd.exe /c "`"$exe`" set $name - --description `"$desc`" < `"$tmp`"" 2>&1
$rc = $LASTEXITCODE
Remove-Item $tmp -Force -ErrorAction SilentlyContinue
$got = (& $exe get $name 2>$null | Out-String).TrimEnd("`r", "`n")
$match = ($got -ceq $want)
if (-not $match) { try { $j = $got | ConvertFrom-Json; $match = ($j.value -ceq $want) } catch {} }
"$Target -> $where kek=$($env:Secrets__MasterKey__FilePath) kek-sha256=$((Get-FileHash $env:Secrets__MasterKey__FilePath).Hash.Substring(0,8)) set rc=$rc out=$(($out | Out-String).Trim()) readback-match=$match"
if ($rc -ne 0 -or -not $match) { $fail++ }
} finally {
Remove-Item $ValueFile, "$ValueFile.nl" -Force -ErrorAction SilentlyContinue; "value file removed: $(-not (Test-Path $ValueFile))"
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue }
}
"SEED DONE failures=$fail"; exit $fail
@@ -0,0 +1,213 @@
#requires -Version 5.1
# PROD STAGE COPY (2026-11-19): mxaccessgw e74918cc scripts/deploy/Install-MxGatewayService.ps1 + 3 patches marked PROD-STAGE PATCH.
<#
.SYNOPSIS
On-box NEW-BOX install of the MxAccess Gateway as an NSSM service (mxaccessgw#220).
Companion of Upgrade-MxGatewayService.ps1; see docs/runbooks/DeployUpgradeNssmService.md.
.DESCRIPTION
RUN ON THE WINDOWS BOX (Windows PowerShell 5.1). Creates the service ONCE, with:
* Application / AppDirectory / DisplayName / Description, auto start, account (default LocalSystem);
* NSSM exit handling: restart on any exit with a throttle, graceful-stop timeouts that cover the
Worker shutdown grace (MxGateway:Worker:ShutdownTimeoutSeconds, default 10 s);
* stdout/stderr redirection with rotation;
* Windows SCM recovery (sc failure: restart, restart, restart + reset period);
* the environment block, set in ONE `nssm set AppEnvironmentExtra` call on a service that has none
(so there is nothing to wipe). The env comes from -EnvFile (KEY=VALUE lines, # comments) and/or
-Env (hashtable); -Env wins. Secret VALUES are never printed (dry-run shows names and masks values).
It does NOT start the service unless -Start is given (review the env first), and it refuses to run if
the service already exists (use Upgrade-MxGatewayService.ps1) or if any gateway process is running
(OrphanWorkerTerminator trap: a second gateway kills live Workers).
Required env (checked by name): ZB_SECRETS_MASTER_KEY, a *Pepper* entry (MxGateway__ApiKeyPepper),
MxGateway__Worker__ExecutablePath (defaults to <WorkerDir>\ZB.MOM.WW.MxGateway.Worker.exe),
MxGateway__Authentication__SqlitePath and Secrets__SqlitePath (default under -DataDir; must be absolute and
OUTSIDE the content root, the validator is fail-closed), ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT.
Kestrel endpoints default to docs/GatewayConfiguration.md's cleartext h2c :5120 + dashboard :5130 unless
the env supplies Kestrel__Endpoints__*.
-DryRun (or -WhatIf) prints every mutating action and performs none.
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[Parameter(Mandatory = $true)][string]$ServerDir,
[Parameter(Mandatory = $true)][string]$WorkerDir,
[string]$ServiceName = 'MxAccessGw',
[string]$NssmPath = 'nssm',
[string]$DataDir = 'C:\ProgramData\MxGateway',
[string]$LogDir,
[string]$EnvFile,
[hashtable]$Env = @{},
[string]$ServerExeName = 'ZB.MOM.WW.MxGateway.Server.exe',
# PROD-STAGE PATCH 1: the wonder Worker build is named MxGateway.Worker.exe, not ZB.MOM.WW.MxGateway.Worker.exe.
[string]$WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe',
[string]$ObjectName = 'LocalSystem',
[string]$ServiceAccountPassword,
[string]$EnvironmentName = 'Production',
[int]$RestartDelayMs = 5000,
[int]$ThrottleMs = 10000,
[int]$StopGraceMs = 20000,
[int]$RotateBytes = 10485760,
[switch]$Start,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
$ServerProc = 'ZB.MOM.WW.MxGateway.Server'
$WorkerProc = 'ZB.MOM.WW.MxGateway.Worker'
function Say([string]$m) { Write-Host $m }
function Fail([string]$m) { throw "ABORT: $m" }
function Step([string]$Description, [scriptblock]$Action) {
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
Say "[ACTION] $Description"; & $Action
}
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
$raw = & $NssmPath @NssmArgs 2>&1; $code = $LASTEXITCODE
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs[0..1] -join ' ') failed (exit $code): $text" }
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
}
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Mask([string]$k, [string]$v) {
# PROD-STAGE PATCH 3: connection strings can carry a SQL password; mask them too.
if ($k -match 'KEY|PASSWORD|SECRET|PEPPER|TOKEN|CONNECTIONSTRING' -and $k -notmatch 'Path$|Url$') { '****' } else { $v }
}
try {
Say "Install-MxGatewayService service=$ServiceName dry=$($script:Dry)"
# --- environment assembly -------------------------------------------------------------------------
$envMap = [ordered]@{}
if ($EnvFile) {
if (-not (Test-Path -LiteralPath $EnvFile)) { Fail "-EnvFile '$EnvFile' not found" }
foreach ($line in (Get-Content -LiteralPath $EnvFile)) {
$t = $line.Trim()
if (-not $t -or $t.StartsWith('#')) { continue }
if ($t -match '^([^=\s]+)=(.*)$') { $envMap[$Matches[1]] = $Matches[2] } else { Fail "-EnvFile line is not KEY=VALUE: '$t'" }
}
}
foreach ($k in $Env.Keys) { $envMap[[string]$k] = [string]$Env[$k] }
$workerExe = Join-Path $WorkerDir $WorkerExeName
if (-not $envMap.Contains('MxGateway__Worker__ExecutablePath')) { $envMap['MxGateway__Worker__ExecutablePath'] = $workerExe }
if (-not $envMap.Contains('MxGateway__Authentication__SqlitePath')) { $envMap['MxGateway__Authentication__SqlitePath'] = Join-Path $DataDir 'gateway-auth.db' }
if (-not $envMap.Contains('Secrets__SqlitePath')) { $envMap['Secrets__SqlitePath'] = Join-Path $DataDir 'mxgateway-secrets.db' }
if (-not $envMap.Contains('ASPNETCORE_ENVIRONMENT')) { $envMap['ASPNETCORE_ENVIRONMENT'] = $EnvironmentName }
if (-not $envMap.Contains('DOTNET_ENVIRONMENT')) { $envMap['DOTNET_ENVIRONMENT'] = $EnvironmentName }
if (-not ($envMap.Keys | Where-Object { $_ -like 'Kestrel__Endpoints__*' })) {
$envMap['Kestrel__Endpoints__Http__Url'] = 'http://0.0.0.0:5120'
$envMap['Kestrel__Endpoints__Http__Protocols'] = 'Http2'
$envMap['Kestrel__Endpoints__Dashboard__Url'] = 'http://0.0.0.0:5130'
Say '[info] no Kestrel__Endpoints__* supplied: using the documented cleartext defaults (5120 h2c, 5130 dashboard)'
}
# --- validation -------------------------------------------------------------------------------------
# PROD-STAGE PATCH 2: accept a file KEK (Secrets__MasterKey__Source=File + FilePath), the QA/PROD pattern.
if ($envMap['Secrets__MasterKey__Source'] -eq 'File') {
$kf = $envMap['Secrets__MasterKey__FilePath']
if (-not $kf -or -not (Test-Rooted $kf)) { Fail 'Secrets__MasterKey__Source=File needs an absolute Secrets__MasterKey__FilePath' }
if (-not $script:Dry -and -not (Test-Path -LiteralPath $kf)) { Fail "KEK file '$kf' not found" }
} elseif (-not $envMap.Contains('ZB_SECRETS_MASTER_KEY') -or -not $envMap['ZB_SECRETS_MASTER_KEY']) { Fail 'env must supply ZB_SECRETS_MASTER_KEY (the Secrets KEK) or Secrets__MasterKey__Source=File' }
if (-not ($envMap.Keys | Where-Object { $_ -match 'Pepper' })) { Fail 'env must supply the API-key pepper (MxGateway__ApiKeyPepper); without it no issued token verifies' }
foreach ($k in 'MxGateway__Authentication__SqlitePath', 'Secrets__SqlitePath') {
$v = $envMap[$k]
if (-not (Test-Rooted $v)) { Fail "$k '$v' must be absolute" }
if ((Test-Inside $v $ServerDir) -or (Test-Inside $v $WorkerDir)) { Fail "$k '$v' is inside the content root; the validator is fail-closed and an upgrade swap would lose it" }
}
foreach ($d in @($ServerDir, $WorkerDir)) { if (-not (Test-Rooted $d)) { Fail "'$d' must be an absolute path" } }
if ((Test-Inside $WorkerDir $ServerDir) -or (Test-Inside $ServerDir $WorkerDir)) { Fail 'Server and Worker directories must be siblings, not nested (the upgrade swap renames them independently)' }
if (-not (Test-Path -LiteralPath (Join-Path $ServerDir $ServerExeName))) { Fail "$ServerExeName not found in $ServerDir" }
if (-not (Test-Path -LiteralPath $workerExe)) { Fail "$WorkerExeName not found in $WorkerDir" }
if ($envMap['MxGateway__Worker__ExecutablePath'] -ne $workerExe) { Say '[WARN] MxGateway__Worker__ExecutablePath differs from <WorkerDir>\Worker.exe; keep ONE canonical spelling' }
if ($envMap['ASPNETCORE_ENVIRONMENT'] -ne 'Production') { Say '[WARN] ASPNETCORE_ENVIRONMENT is not exactly Production: DisableLogin / plaintext-LDAP hard-stops are only enforced for Production' }
# --- one-gateway guard + existence -------------------------------------------------------------------
$st = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
if ($st.Code -eq 0) { Fail "service '$ServiceName' already exists (state $($st.Text)); use Upgrade-MxGatewayService.ps1" }
# Both naming schemes (ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes) plus the exe being installed (#230).
$gwImages = @($ServerProc, $WorkerProc, 'MxGateway.Server', 'MxGateway.Worker', [IO.Path]::GetFileNameWithoutExtension($ServerExeName)) | Select-Object -Unique
$procs = @(Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.ProcessName -in $gwImages })
if ($procs.Count -gt 0) { Fail "gateway process(es) already running (PIDs $(($procs.Id) -join ',')); installing/starting a second gateway kills their Workers" }
if (-not $LogDir) { $LogDir = Join-Path $DataDir 'logs' }
$exe = Join-Path $ServerDir $ServerExeName
Say "[info] env entries ($($envMap.Count)):"
foreach ($k in $envMap.Keys) { Say " $k=$(Mask $k $envMap[$k])" }
# --- install ------------------------------------------------------------------------------------------------
Step "create $DataDir and $LogDir (stores/certs/logs live OUTSIDE the content root)" {
New-Item -ItemType Directory -Force -Path $DataDir, $LogDir | Out-Null
}
Step "nssm install $ServiceName `"$exe`"" { [void](Invoke-Nssm -NssmArgs @('install', $ServiceName, $exe)); $script:Installed = $true }
$settings = @(
@('AppDirectory', $ServerDir),
@('DisplayName', 'MxAccess Gateway'),
@('Description', 'MxAccess Gateway (gRPC) - Server; launches the x86 Worker per session'),
@('Start', 'SERVICE_AUTO_START'),
@('Type', 'SERVICE_WIN32_OWN_PROCESS'),
@('AppExit', 'Default', 'Restart'),
@('AppRestartDelay', "$RestartDelayMs"),
@('AppThrottle', "$ThrottleMs"),
@('AppStopMethodConsole', "$StopGraceMs"),
@('AppStopMethodWindow', '0'),
@('AppStopMethodThreads', '0'),
@('AppStdout', (Join-Path $LogDir 'service-stdout.log')),
@('AppStderr', (Join-Path $LogDir 'service-stderr.log')),
@('AppRotateFiles', '1'),
@('AppRotateOnline', '1'),
@('AppRotateBytes', "$RotateBytes")
)
foreach ($s in $settings) {
Step "nssm set $ServiceName $($s -join ' ')" { [void](Invoke-Nssm -NssmArgs (@('set', $ServiceName) + $s)) }
}
if ($ObjectName -and $ObjectName -ne 'LocalSystem') {
Step "nssm set $ServiceName ObjectName $ObjectName <password>" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', $ObjectName, $ServiceAccountPassword)) }
} else {
Step "nssm set $ServiceName ObjectName LocalSystem" { [void](Invoke-Nssm -NssmArgs @('set', $ServiceName, 'ObjectName', 'LocalSystem')) }
}
# Env block: first and only AppEnvironmentExtra write; the service has none, so nothing is wiped.
$pairs = @($envMap.Keys | ForEach-Object { "$_=$($envMap[$_])" })
Step "nssm set $ServiceName AppEnvironmentExtra <$($envMap.Count) entries, values hidden>" {
[void](Invoke-Nssm -NssmArgs (@('set', $ServiceName, 'AppEnvironmentExtra') + $pairs))
}
# SCM recovery: restart x3 (5 s, 5 s, 30 s), reset the counter after 1 day; also on non-crash exits.
Step "sc.exe failure $ServiceName reset= 86400 actions= restart/5000/restart/5000/restart/30000" {
& sc.exe failure $ServiceName reset= 86400 actions= 'restart/5000/restart/5000/restart/30000' | Out-Null
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failure exited $LASTEXITCODE" }
}
Step "sc.exe failureflag $ServiceName 1" {
& sc.exe failureflag $ServiceName 1 | Out-Null
if ($LASTEXITCODE -ne 0) { Fail "sc.exe failureflag exited $LASTEXITCODE" }
}
# Verify read-back (names only) so a silent miss is caught before the first start.
Step "verify: nssm get AppEnvironmentExtra has $($envMap.Count) entries" {
$txt = (Invoke-Nssm -NssmArgs @('get', $ServiceName, 'AppEnvironmentExtra')).Text
$n = @($txt -split "[`r`n]+" | Where-Object { $_.Trim() -match '^[^=\s]+=' }).Count
if ($n -ne $envMap.Count) { Fail "env read-back has $n entries, expected $($envMap.Count)" }
}
if ($Start) {
Step "nssm start $ServiceName (exactly once)" { [void](Invoke-Nssm -NssmArgs @('start', $ServiceName)) }
Say 'Next: GET /health/ready on the dashboard port, then prove a Worker session with an API key (runbook section 7).'
} else {
Say '[info] service installed STOPPED. Review the env, then `nssm start` (or re-run with -Start). Seed secrets with the `secret` CLI using the SAME Secrets__SqlitePath + ZB_SECRETS_MASTER_KEY.'
}
Say '== DONE =='
exit 0
} catch {
Write-Host "ERROR: $($_.Exception.Message)" -ForegroundColor Red
if ($script:Installed) { Write-Host "PARTIAL INSTALL: inspect, then `nssm remove $ServiceName confirm` and re-run." -ForegroundColor Yellow }
exit 1
}
@@ -0,0 +1,869 @@
#requires -Version 5.1
<#
.SYNOPSIS
On-box upgrade / rollback of the NSSM-wrapped MxAccess Gateway (Server + x86 Worker).
Implements docs/runbooks/DeployUpgradeNssmService.md sections 3, 5, 6, 7 and 8 (mxaccessgw#220, #230).
.DESCRIPTION
RUN THIS ON THE WINDOWS BOX ITSELF (Windows PowerShell 5.1 is enough). The new Server publish and the
new x86 Worker build must already be staged on the box (UNC/SMB copy; build both from the same SHA).
Upgrade flow: preflight -> stop (STOP_PENDING handled) -> backup -> swap Server + Worker -> start ->
verify (/health/ready, VersionInfo SHA, service stable, env block unchanged) -> automatic rollback if
start/verify fails. Rollback flow (-Rollback): stop -> swap the .bak directories back -> optional store
restore -> start -> verify.
Safety properties:
* Exactly one gateway per box. The gateway and Worker image names are taken from the INSTALLED exes
(NSSM Application and env MxGateway__Worker__ExecutablePath) plus both known naming schemes
(ZB.MOM.WW.MxGateway.* on windev, MxGateway.* on the wonder boxes), and processes are also matched by
exact path. The guards FAIL CLOSED: a RUNNING service whose gateway process cannot be found is an
abort, not a pass (mxaccessgw#230: the guards used to pass vacuously on the wonder boxes).
* Never writes the NSSM env block (AppEnvironmentExtra), Application or AppDirectory: the swap is
rename-in-place. The env block is fingerprinted (entry names + SHA-256 prefix of each value; values are
never printed) at preflight, after the stop, after the swap, after the start and after the stability
window. Any added, removed or changed entry aborts (and triggers the automatic rollback).
* Live-only top-level files in Server\ and Worker\ are carried into the new directories (appsettings*.json,
dashboard.pfx and other certificates/keys always keep the LIVE copy), hash-verified.
* API-key DB, Secrets store and a file-based master key must live outside the swapped directories; they are
only COPIED (db + -wal + -shm) into *.bak-<stamp> siblings while the service is stopped.
* A self-contained Server is refused if its .NET runtime is older than the live one (no silent downgrade).
-DryRun (or -WhatIf) prints every mutating action and performs none. Read-only probing (service state,
registry env block, process list, disk space, file hashes) still runs, so the guards are exercised for real.
.PARAMETER ServerSource Staged Server publish output (must contain the same exe name as NSSM Application).
.PARAMETER WorkerSource Staged x86 Worker output (must contain the exe name of MxGateway__Worker__ExecutablePath).
.PARAMETER ServiceName NSSM service name. Default MxAccessGw.
.PARAMETER NssmPath nssm executable. Default 'nssm' (on PATH).
.PARAMETER ServerDir Live Server directory. Default: NSSM AppDirectory.
.PARAMETER WorkerDir Live Worker directory. Default: parent of env MxGateway__Worker__ExecutablePath.
.PARAMETER ExpectedSha Short/long git SHA. Staged Server+Worker (preflight) and the deployed Server (verify)
must carry it in ProductVersion ('0.1.2+<sha>'). Builds from 2026-07-09..08-10 have a bogus stamp.
.PARAMETER Label Free-text suffix for backup names (e.g. 'planwrites').
.PARAMETER HealthBaseUrl Default http://localhost:5130 (dashboard port; /health/ready, /healthz anonymous).
.PARAMETER Rollback Roll back instead of upgrading. Uses the newest Server.bak-* / Server.bak.* unless -BackupStamp.
.PARAMETER BackupStamp Stamp of the backup set to roll back to (yyyyMMdd-HHmmss, or yyyyMMddTHHmmss for .bak.<stamp> sets).
.PARAMETER RestoreStores With -Rollback: also restore gateway-auth.db / secrets db from *.bak-<stamp> (schema-bump case).
.PARAMETER NoAutoRollback Do not roll back automatically when start/verification fails.
.PARAMETER TakeNewAppSettings Do not carry the live appsettings*.json over into the new Server dir.
.PARAMETER AllowRuntimeDowngrade Accept a staged self-contained runtime older than the live one.
.PARAMETER SkipFreeSpaceCheck / MinFreeGB / StopTimeoutSeconds / StableSeconds / ReadyTimeoutSeconds tunables.
.PARAMETER DryRun Print, do not act. -WhatIf is equivalent.
#>
[CmdletBinding(SupportsShouldProcess = $true)]
param(
[string]$ServerSource,
[string]$WorkerSource,
[string]$ServiceName = 'MxAccessGw',
[string]$NssmPath = 'nssm',
[string]$ServerDir,
[string]$WorkerDir,
[string]$ExpectedSha,
[string]$Label,
[string]$HealthBaseUrl = 'http://localhost:5130',
[switch]$Rollback,
[string]$BackupStamp,
[switch]$RestoreStores,
[switch]$NoAutoRollback,
[switch]$TakeNewAppSettings,
[switch]$AllowRuntimeDowngrade,
[switch]$SkipFreeSpaceCheck,
[double]$MinFreeGB = 2,
[int]$StopTimeoutSeconds = 30,
[int]$StableSeconds = 60,
[int]$ReadyTimeoutSeconds = 120,
[switch]$DryRun
)
$ErrorActionPreference = 'Stop'
$script:Dry = $DryRun.IsPresent -or $WhatIfPreference
# -WhatIf is handled by $script:Dry (Step). Leaving the preference on makes Windows PowerShell 5.1 cmdlets used for READ-ONLY
# probing misbehave (Get-FileHash returns nothing under -WhatIf), so the plan would abort; found on windev, #230.
$WhatIfPreference = $false
$script:Stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
# Both naming schemes seen in the field: windev runs ZB.MOM.WW.MxGateway.*.exe, the wonder boxes MxGateway.*.exe.
# The installed exe names (NSSM Application, MxGateway__Worker__ExecutablePath) are added to these in Get-Context.
$script:KnownServerImages = @('ZB.MOM.WW.MxGateway.Server', 'MxGateway.Server')
$script:KnownWorkerImages = @('ZB.MOM.WW.MxGateway.Worker', 'MxGateway.Worker')
$script:ServerImages = $script:KnownServerImages
$script:WorkerImages = $script:KnownWorkerImages
$script:EnvWorkerExe = 'MxGateway__Worker__ExecutablePath'
# Live copy always wins for these (host config, certificates, keys), even when the staged build ships one.
$script:PreservePatterns = @('appsettings*.json', '*.pfx', '*.p12', '*.pem', '*.key', '*.crt', '*.cer')
# Never carried from the old build (they are build output; a live-only binary is a leftover of an older build).
$script:BinaryExtensions = @('.dll', '.exe', '.pdb', '.so', '.dylib', '.winmd')
$script:Notes = New-Object System.Collections.Generic.List[string]
$script:Redact = New-Object System.Collections.Generic.List[string]
$script:LockStream = $null
$script:BackupDir = $null
$script:EnvWarnOnly = $false
# --- output (secret values are redacted from every line) ----------------------------------------------------
function Hide([string]$m) {
if ($null -eq $m) { return '' }
foreach ($s in $script:Redact) { if ($s) { $m = $m.Replace($s, '<redacted>') } }
$m
}
function Say([string]$m) { $m = Hide $m; Write-Host $m; $script:Notes.Add($m) }
function Info([string]$m) { Say "[info] $m" }
function Warn([string]$m) { $m = Hide $m; Write-Warning $m; $script:Notes.Add("[WARN] $m") }
function Fail([string]$m) { throw "ABORT: $(Hide $m)" }
# Mutating step: print in dry-run, run otherwise.
function Step([string]$Description, [scriptblock]$Action) {
if ($script:Dry) { Say "[DRYRUN] $Description"; return }
Say "[ACTION] $Description"
& $Action
}
# --- environment seams (the dry-run harness in scripts/deploy/tests replaces these) ---------------------------
function Test-IsWindowsHost { $env:OS -eq 'Windows_NT' }
function Invoke-Nssm {
param([string[]]$NssmArgs, [switch]$AllowFail)
$raw = & $NssmPath @NssmArgs 2>&1
$code = $LASTEXITCODE
$text = (($raw | ForEach-Object { "$_" }) -join "`n") -replace "`0", ''
if ($code -ne 0 -and -not $AllowFail) { Fail "nssm $($NssmArgs -join ' ') failed (exit $code): $text" }
[pscustomobject]@{ Code = $code; Text = $text.Trim() }
}
function Get-NssmValue([string]$Name) { (Invoke-Nssm -NssmArgs @('get', $ServiceName, $Name)).Text }
function Get-ServiceState {
$r = Invoke-Nssm -NssmArgs @('status', $ServiceName) -AllowFail
if ($r.Code -ne 0) { return 'MISSING' }
($r.Text -split "`n")[0].Trim()
}
# The env block exactly as NSSM stores it (REG_MULTI_SZ), one 'NAME=value' per element. Falls back to nssm get.
function Read-EnvBlockLines {
$key = "HKLM:\SYSTEM\CurrentControlSet\Services\$ServiceName\Parameters"
if ((Test-IsWindowsHost) -and (Test-Path -LiteralPath $key)) {
$p = Get-ItemProperty -LiteralPath $key -Name AppEnvironmentExtra -ErrorAction SilentlyContinue
if ($p) { return @($p.AppEnvironmentExtra | Where-Object { $_ }) }
return @()
}
@((Get-NssmValue 'AppEnvironmentExtra') -split "[`r`n]+" | Where-Object { $_.Trim() })
}
# Id, ParentId, Name (image name without .exe), Path (may be $null when unreadable).
function Get-ProcessTable {
if (Get-Command Get-CimInstance -ErrorAction SilentlyContinue) {
return @(Get-CimInstance Win32_Process | ForEach-Object {
[pscustomobject]@{ Id = [int]$_.ProcessId; ParentId = [int]$_.ParentProcessId; Name = [IO.Path]::GetFileNameWithoutExtension([string]$_.Name); Path = $_.ExecutablePath }
})
}
@(Get-Process -ErrorAction SilentlyContinue | ForEach-Object {
$p = $null; try { $p = $_.Path } catch { }
[pscustomobject]@{ Id = $_.Id; ParentId = 0; Name = $_.ProcessName; Path = $p }
})
}
# PID of the NSSM service host (the gateway Server is its child). $null when it cannot be read.
function Get-ServiceHostPid {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
$s = Get-CimInstance Win32_Service -Filter "Name='$ServiceName'" -ErrorAction SilentlyContinue
if ($s -and $s.ProcessId) { return [int]$s.ProcessId }
$null
}
# Other services whose command line launches the gateway exe. $null when the check is unavailable.
function Get-OtherGatewayServices([string]$ExeName) {
if (-not (Get-Command Get-CimInstance -ErrorAction SilentlyContinue)) { return $null }
# The leading comma keeps an EMPTY result an array: a bare @() reaches the caller as $null ('check unavailable').
, @(Get-CimInstance Win32_Service | Where-Object { $_.Name -ne $ServiceName -and $_.PathName -and $_.PathName -like "*$ExeName*" } | ForEach-Object { $_.Name })
}
function Stop-ProcessById([int]$Id) { Stop-Process -Id $Id -Force -ErrorAction SilentlyContinue }
function Get-HttpStatus([string]$Url) {
try { return [int](Invoke-WebRequest -Uri $Url -UseBasicParsing -TimeoutSec 10).StatusCode }
catch { if ($_.Exception.Response) { return [int]$_.Exception.Response.StatusCode }; return 0 }
}
function Get-ProductSha([string]$Exe) {
try {
$pv = [System.Diagnostics.FileVersionInfo]::GetVersionInfo($Exe).ProductVersion
if ($pv -match '\+([0-9a-fA-F]{7,40})(\b|$)') { return $Matches[1].ToLowerInvariant() }
} catch { }
$null
}
# 'X86', 'Amd64', 'MSIL', ... or $null when it cannot be read.
function Get-ExeArchitecture([string]$Exe) {
try { return "$([Reflection.AssemblyName]::GetAssemblyName($Exe).ProcessorArchitecture)" } catch { return $null }
}
# Restrict a directory to SYSTEM + Administrators, inheritance off. Throws if that fails.
function Protect-Directory([string]$Path) {
if (-not (Test-IsWindowsHost)) { Warn "not Windows: $Path is not ACL-restricted"; return }
$null = & icacls.exe $Path /inheritance:r /grant:r '*S-1-5-18:(OI)(CI)F' '*S-1-5-32-544:(OI)(CI)F' 2>&1
if ($LASTEXITCODE -ne 0) { Fail "icacls could not restrict $Path (exit $LASTEXITCODE); refusing to write the env block there" }
}
function Export-ServiceKey([string]$Destination) {
if (-not (Test-IsWindowsHost)) { Info 'not Windows: service registry export skipped'; return }
$null = & reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$ServiceName" $Destination /y 2>&1
if ($LASTEXITCODE -ne 0) { Fail "reg export of the service key failed (exit $LASTEXITCODE)" }
}
function Get-DirSddl([string]$Path) {
if (-not (Test-IsWindowsHost)) { return $null }
try { (Get-Acl -LiteralPath $Path).Sddl } catch { $null }
}
# Give a carried file the old file's ACL when the old one had explicit (non-inherited) entries.
function Copy-ExplicitAcl([string]$From, [string]$To) {
if (-not (Test-IsWindowsHost)) { return }
try {
$a = Get-Acl -LiteralPath $From
if (@($a.Access | Where-Object { -not $_.IsInherited }).Count -gt 0 -or $a.AreAccessRulesProtected) {
Set-Acl -LiteralPath $To -AclObject $a
Info " ACL of $(Split-Path -Leaf $From) copied (it had explicit entries)"
}
} catch { Warn "could not copy the ACL onto ${To}: $($_.Exception.Message)" }
}
# --- pure helpers ---------------------------------------------------------------------------------------------
function Norm([string]$p) { ($p -replace '\\', '/').TrimEnd('/') }
function Test-Rooted([string]$p) { $p -match '^[A-Za-z]:[\\/]' -or $p.StartsWith('/') -or $p.StartsWith('\\') }
function Test-Inside([string]$Path, [string]$Root) {
if (-not $Path -or -not $Root) { return $false }
$a = (Norm $Path).ToLowerInvariant(); $b = (Norm $Root).ToLowerInvariant()
$a -eq $b -or $a.StartsWith("$b/")
}
function Test-SamePath([string]$A, [string]$B) { $A -and $B -and ((Norm $A).ToLowerInvariant() -eq (Norm $B).ToLowerInvariant()) }
function Get-Leaf([string]$p) { if (-not $p) { return $null }; ((Norm $p) -split '/')[-1] }
function Get-Stem([string]$p) { $l = Get-Leaf $p; if (-not $l) { return $null }; [IO.Path]::GetFileNameWithoutExtension($l) }
function Test-ShaMatch([string]$Actual, [string]$Expected) {
if (-not $Actual -or -not $Expected) { return $false }
$e = $Expected.ToLowerInvariant()
$Actual.StartsWith($e) -or $e.StartsWith($Actual)
}
function Get-TextHashPrefix([string]$Text) {
$sha = [Security.Cryptography.SHA256]::Create()
try { (($sha.ComputeHash([Text.Encoding]::UTF8.GetBytes($Text)) | ForEach-Object { $_.ToString('x2') }) -join '').Substring(0, 12) }
finally { $sha.Dispose() }
}
function Get-FileSha([string]$Path) {
$sha = [Security.Cryptography.SHA256]::Create(); $fs = [IO.File]::OpenRead($Path)
try { (($sha.ComputeHash($fs) | ForEach-Object { $_.ToString('X2') }) -join '') } finally { $fs.Dispose(); $sha.Dispose() }
}
function Test-SensitiveName([string]$Name) { ($Name -match 'KEY|PEPPER|PASS|SECRET|TOKEN|PWD|CREDENTIAL') -and ($Name -notmatch '(Path|Source|EnvVarName|Name)$') }
# name -> value. Every sensitive value is registered for redaction.
function ConvertTo-EnvMap([string[]]$Lines) {
$map = [ordered]@{}
foreach ($line in $Lines) {
$i = $line.IndexOf('=')
if ($i -le 0) { continue }
$k = $line.Substring(0, $i).Trim(); $v = $line.Substring($i + 1)
$map[$k] = $v
if ((Test-SensitiveName $k) -and $v.Length -ge 4 -and -not $script:Redact.Contains($v)) { $script:Redact.Add($v) }
}
$map
}
# Names + per-value SHA-256 prefixes. Never carries a value.
function Get-EnvFingerprint([string[]]$Lines) {
$map = ConvertTo-EnvMap $Lines
$entries = [ordered]@{}
foreach ($k in $map.Keys) { $entries[$k] = Get-TextHashPrefix ([string]$map[$k]) }
[pscustomobject]@{ Count = @($Lines).Count; Names = @($map.Keys); Entries = $entries; BlockHash = (Get-TextHashPrefix (@($Lines) -join "`n")) }
}
# List of differences (names only). Empty when identical.
function Compare-EnvFingerprint($Before, $After) {
$d = New-Object System.Collections.Generic.List[string]
foreach ($k in $Before.Entries.Keys) {
if (-not $After.Entries.Contains($k)) { $d.Add("removed $k") }
elseif ($After.Entries[$k] -ne $Before.Entries[$k]) { $d.Add("changed $k (sha256 $($Before.Entries[$k]) -> $($After.Entries[$k]))") }
}
foreach ($k in $After.Entries.Keys) { if (-not $Before.Entries.Contains($k)) { $d.Add("added $k") } }
if ($d.Count -eq 0 -and $Before.BlockHash -ne $After.BlockHash) { $d.Add("block hash $($Before.BlockHash) -> $($After.BlockHash) (order, duplicates or blank entries changed)") }
, $d.ToArray()
}
function Format-EnvFingerprint($Fp) { "$($Fp.Count) entries, block sha256 $($Fp.BlockHash): " + (($Fp.Entries.Keys | ForEach-Object { "$_=$($Fp.Entries[$_])" }) -join ', ') }
# Re-read the env block and abort on any difference from the preflight fingerprint.
function Assert-EnvUnchanged($ctx, [string]$When) {
$now = Get-EnvFingerprint (Read-EnvBlockLines)
$diff = Compare-EnvFingerprint $ctx.EnvFp $now
if ($diff.Count -gt 0) {
# During an automatic rollback the difference is already the reported failure; restoring the old build still goes ahead.
if ($script:EnvWarnOnly) { Warn "NSSM env block changed ${When}: $($diff -join '; ') (not written by this script; restore it from the backup dir's AppEnvironmentExtra.txt if unintended)"; return }
Fail "NSSM env block changed ${When}: $($diff -join '; ')"
}
Info "env block unchanged $When ($($now.Count) entries, block sha256 $($now.BlockHash))"
}
# Directory listings via .NET (FileInfo/DirectoryInfo, hidden files included). Get-ChildItem costs ~250 ms per call
# under pwsh on macOS (per-file owner lookups), which made the dry-run harness 20x slower; on Windows both are fine.
function Get-Files([string]$Dir, [string]$Filter = '*', [switch]$Recurse) {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
$o = if ($Recurse) { [IO.SearchOption]::AllDirectories } else { [IO.SearchOption]::TopDirectoryOnly }
try { (New-Object IO.DirectoryInfo $Dir).GetFiles($Filter, $o) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-Dirs([string]$Dir, [string]$Filter = '*') {
if (-not $Dir -or -not [IO.Directory]::Exists($Dir)) { return }
try { (New-Object IO.DirectoryInfo $Dir).GetDirectories($Filter) } catch { Warn "cannot list ${Dir}: $($_.Exception.Message)" }
}
function Get-StoreFiles([string]$Db) { @($Db, "$Db-wal", "$Db-shm") | Where-Object { $_ -and (Test-Path -LiteralPath $_) } }
function Get-DirSize([string]$p) {
if (-not (Test-Path -LiteralPath $p)) { return 0 }
$s = (@(Get-Files $p -Recurse) | Measure-Object Length -Sum).Sum
if ($s) { [double]$s } else { 0 }
}
# Version of a self-contained .NET runtime in a directory (from mscordaccore_<arch>_<arch>_<ver>.dll), or $null (framework-dependent).
function Get-BundledRuntimeVersion([string]$Dir) {
$f = @(Get-Files $Dir 'mscordaccore_*.dll')
foreach ($x in $f) { if ($x.Name -match '_(\d+\.\d+\.\d+(\.\d+)?)\.dll$') { return [version]$Matches[1] } }
$null
}
# --- processes --------------------------------------------------------------------------------------------------
function Test-GatewayServer($p, $ctx) { ($script:ServerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.Application) }
function Test-GatewayWorker($p, $ctx) { ($script:WorkerImages -contains $p.Name) -or (Test-SamePath $p.Path $ctx.WorkerExe) }
function Get-GatewayProcesses($ctx) {
@(Get-ProcessTable | Where-Object { (Test-GatewayServer $_ $ctx) -or (Test-GatewayWorker $_ $ctx) } | ForEach-Object {
$role = if (Test-GatewayServer $_ $ctx) { 'Server' } else { 'Worker' }
$_ | Add-Member -NotePropertyName Role -NotePropertyValue $role -PassThru -Force
})
}
function Format-Procs($procs) { (@($procs) | ForEach-Object { "$($_.Name)#$($_.Id)<-$($_.ParentId) [$(if ($_.Path) { $_.Path } else { 'path unreadable' })]" }) -join ', ' }
function Assert-NoGatewayProcess($ctx, [string]$When) {
$left = @(Get-GatewayProcesses $ctx)
if ($left.Count -gt 0) { Fail "${When}: gateway process(es) still alive: $(Format-Procs $left)" }
}
# The one-gateway guard. Fails closed: a RUNNING service whose gateway cannot be seen is an abort.
function Assert-SingleGateway($ctx, [string]$State) {
$procs = @(Get-GatewayProcesses $ctx)
$servers = @($procs | Where-Object { $_.Role -eq 'Server' })
$workers = @($procs | Where-Object { $_.Role -eq 'Worker' })
Info "gateway processes on the box (images $($script:ServerImages -join '|') / $($script:WorkerImages -join '|')): $(if ($procs.Count) { Format-Procs $procs } else { 'none' })"
if ($servers.Count -gt 1) { Fail "$($servers.Count) gateway Server processes are running ($(Format-Procs $servers)). A second gateway is the OrphanWorkerTerminator trap - resolve manually." }
switch ($State) {
'SERVICE_STOPPED' {
if ($procs.Count -gt 0) { Fail "service is STOPPED but gateway process(es) exist ($(Format-Procs $procs)): a rogue second gateway or an unclean stop. Starting would terminate those workers." }
}
'SERVICE_RUNNING' {
if ($servers.Count -eq 0) {
Fail "service is RUNNING but no gateway Server process was found (looked for image names $($script:ServerImages -join ', ') and path $($ctx.Application)). The process guards cannot see this gateway, so they would pass vacuously; refusing."
}
$s = $servers[0]
if ($s.Path -and -not (Test-SamePath $s.Path $ctx.Application)) { Fail "the running gateway is $($s.Path), not the service's Application $($ctx.Application): a second gateway is running and the service's own is not" }
if (-not $s.Path) { Warn "gateway Server #$($s.Id) path is unreadable; matched by image name only" }
$hostPid = Get-ServiceHostPid
if ($hostPid) {
if ($s.ParentId -ne $hostPid) { Fail "gateway Server #$($s.Id) is not a child of the $ServiceName service host #${hostPid} (parent #$($s.ParentId)): it is another gateway" }
Info "gateway Server #$($s.Id) is the service's own (child of service host #$hostPid)"
} else { Warn 'service host PID unavailable; parent check skipped' }
$foreign = @($workers | Where-Object { $_.ParentId -and $_.ParentId -ne $s.Id })
if ($foreign.Count -gt 0) { Warn "Worker process(es) not owned by the service's gateway: $(Format-Procs $foreign). They are killed as orphans on the next start." }
$wrongPath = @($workers | Where-Object { $_.Path -and $ctx.WorkerExe -and -not (Test-SamePath $_.Path $ctx.WorkerExe) })
if ($wrongPath.Count -gt 0) { Warn "Worker process(es) not running the configured $($ctx.WorkerExe): $(Format-Procs $wrongPath)" }
}
'SERVICE_STOP_PENDING' { Warn 'service is STOP_PENDING; the stop step will finish it' }
default {
if ($servers.Count -gt 0) { Fail "a gateway Server process exists while the service is $State" }
}
}
}
# --- context discovery (read only) -------------------------------------------------------------------------
function Get-Context {
$state = Get-ServiceState
if ($state -eq 'MISSING') { Fail "service '$ServiceName' does not exist (use Install-MxGatewayService.ps1 for a new box)" }
$ctx = [ordered]@{}
$ctx.State = $state
$ctx.Application = Get-NssmValue 'Application'
$ctx.AppDirectory = Get-NssmValue 'AppDirectory'
$lines = @(Read-EnvBlockLines)
$ctx.Env = ConvertTo-EnvMap $lines
$ctx.EnvFp = Get-EnvFingerprint $lines
$ctx.EnvCount = $ctx.EnvFp.Count
if ($ServerDir) { $ctx.ServerDir = $ServerDir } else { $ctx.ServerDir = $ctx.AppDirectory }
$wexe = $ctx.Env[$script:EnvWorkerExe]
$ctx.WorkerExe = $wexe
if ($WorkerDir) { $ctx.WorkerDir = $WorkerDir }
elseif ($wexe) { $ctx.WorkerDir = Split-Path -Parent $wexe }
else { Fail "cannot determine the Worker directory: env $($script:EnvWorkerExe) is not in the NSSM env block; pass -WorkerDir" }
$pd = 'C:\ProgramData\MxGateway'
$auth = $ctx.Env['MxGateway__Authentication__SqlitePath']; if (-not $auth) { $auth = "$pd\gateway-auth.db" }
$sec = $ctx.Env['Secrets__SqlitePath']; if (-not $sec) { $sec = "$pd\mxgateway-secrets.db" }
$ctx.AuthDb = $auth; $ctx.SecretsDb = $sec
$ctx.KekFile = $ctx.Env['Secrets__MasterKey__FilePath']
$ctx.ServerExeName = Get-Leaf $ctx.Application
$ctx.WorkerExeName = Get-Leaf $wexe
if (-not $ctx.WorkerExeName) { $ctx.WorkerExeName = 'ZB.MOM.WW.MxGateway.Worker.exe' }
# Process image names: the installed ones first, then both known schemes.
$script:ServerImages = @(@(Get-Stem $ctx.Application) + $script:KnownServerImages | Where-Object { $_ } | Select-Object -Unique)
$script:WorkerImages = @(@(Get-Stem $wexe) + $script:KnownWorkerImages | Where-Object { $_ } | Select-Object -Unique)
[pscustomobject]$ctx
}
# --- preflight -----------------------------------------------------------------------------------------------
function Test-MasterKey($ctx) {
$src = $ctx.Env['Secrets__MasterKey__Source']
$envVar = $ctx.Env['Secrets__MasterKey__EnvVarName']; if (-not $envVar) { $envVar = 'ZB_SECRETS_MASTER_KEY' }
if ($ctx.KekFile -or $src -eq 'File') {
if (-not $ctx.KekFile) { Fail 'Secrets__MasterKey__Source=File but no Secrets__MasterKey__FilePath in the env block' }
if (-not (Test-Rooted $ctx.KekFile)) { Fail "master key file '$($ctx.KekFile)' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) { if (Test-Inside $ctx.KekFile $root) { Fail "master key file '$($ctx.KekFile)' is INSIDE '$root' - the swap would lose it" } }
if (Test-Path -LiteralPath $ctx.KekFile) { Info "master key: file $($ctx.KekFile) (present, outside the swap)" } else { Fail "master key file '$($ctx.KekFile)' does not exist (the Secrets store would be unreadable)" }
} elseif ($ctx.Env.Contains($envVar)) {
Info "master key: env block entry $envVar"
} else {
Warn "the env block carries no master key ($envVar or Secrets__MasterKey__FilePath); it comes from elsewhere (appsettings or the account environment). The upgrade does not change that, but confirm the Secrets store opens after the start."
}
}
function Invoke-Preflight($ctx) {
Say '== Preflight =='
Info "service '$ServiceName' state: $($ctx.State)"
Info "Application : $($ctx.Application)"
Info "AppDirectory: $($ctx.AppDirectory)"
Info "Server dir : $($ctx.ServerDir)"
Info "Worker dir : $($ctx.WorkerDir) (exe env: $($ctx.WorkerExe))"
Info "API-key DB : $($ctx.AuthDb)"
Info "Secrets DB : $($ctx.SecretsDb)"
Info "NSSM env block fingerprint (names + value sha256 prefixes, no values): $(Format-EnvFingerprint $ctx.EnvFp)"
if ($ctx.EnvCount -eq 0) { Fail 'NSSM env block is empty - refusing: it holds Kestrel/store paths, the master key and the pepper' }
if (-not ($ctx.Env.Keys | Where-Object { $_ -match 'Pepper' })) {
Warn 'no *Pepper* entry in the env block; confirm the API-key pepper is supplied another way or issued tokens will not verify'
}
if ($ctx.Env['ASPNETCORE_ENVIRONMENT'] -ne 'Production' -and $ctx.Env['DOTNET_ENVIRONMENT'] -ne 'Production') {
Warn 'neither ASPNETCORE_ENVIRONMENT nor DOTNET_ENVIRONMENT is exactly Production (runbook trap 6) - do not change it during an upgrade'
}
if (-not (Test-SamePath (Split-Path -Parent $ctx.Application) $ctx.ServerDir)) { Fail "NSSM Application '$($ctx.Application)' is not inside the Server dir '$($ctx.ServerDir)'" }
if ($ctx.WorkerExe -and -not (Test-SamePath (Split-Path -Parent $ctx.WorkerExe) $ctx.WorkerDir)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' is not inside the Worker dir '$($ctx.WorkerDir)'" }
# State that must survive: rooted and outside the swapped directories.
foreach ($pair in @(@('API-key DB', $ctx.AuthDb), @('Secrets DB', $ctx.SecretsDb))) {
if (-not (Test-Rooted $pair[1])) { Fail "$($pair[0]) path '$($pair[1])' is not absolute" }
foreach ($root in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (Test-Inside $pair[1] $root) { Fail "$($pair[0]) '$($pair[1])' is INSIDE '$root' - the swap would lose it. Move it out first (runbook section 1)." }
}
if (Test-Path -LiteralPath $pair[1]) { Info "$($pair[0]) present ($((Get-StoreFiles $pair[1]).Count) file(s) incl. wal/shm)" }
else { Warn "$($pair[0]) '$($pair[1])' does not exist yet (fresh store will be created on start)" }
}
Test-MasterKey $ctx
# Directories
foreach ($d in @($ctx.ServerDir, $ctx.WorkerDir)) {
if (-not (Test-Path -LiteralPath $d -PathType Container)) { Fail "live directory '$d' does not exist" }
}
if ((Test-Inside $ctx.WorkerDir $ctx.ServerDir) -or (Test-Inside $ctx.ServerDir $ctx.WorkerDir)) {
Fail 'Server and Worker directories are nested; the rename-in-place swap needs sibling directories'
}
if (-not (Test-Path -LiteralPath $ctx.Application)) { Fail "NSSM Application '$($ctx.Application)' does not exist" }
if ($ctx.WorkerExe -and -not (Test-Path -LiteralPath $ctx.WorkerExe)) { Fail "$($script:EnvWorkerExe) '$($ctx.WorkerExe)' does not exist" }
$dbs = @(@(Get-Files $ctx.ServerDir -Recurse) + @(Get-Files $ctx.WorkerDir -Recurse) | Where-Object { $_.Name -match '\.db(-wal|-shm)?$' })
if ($dbs.Count -gt 0) { Fail "SQLite file(s) inside the swapped directories would be orphaned by the rename: $(($dbs | ForEach-Object { $_.FullName }) -join ', ')" }
# One-gateway guard (fail closed)
Assert-SingleGateway $ctx $ctx.State
$others = Get-OtherGatewayServices $ctx.ServerExeName
if ($null -eq $others) { Info 'Win32_Service cross-check unavailable on this host (skipped)' }
elseif (@($others).Count -gt 0) { Fail "another service ($(@($others) -join ', ')) also launches $($ctx.ServerExeName); two gateways on one box is forbidden" }
else { Info "no other service launches $($ctx.ServerExeName)" }
if ($Rollback) { Say '== Preflight OK =='; return }
# Sources
if (-not $ServerSource -or -not $WorkerSource) { Fail '-ServerSource and -WorkerSource are required for an upgrade' }
$stagedServerExe = Join-Path $ServerSource $ctx.ServerExeName
$stagedWorkerExe = Join-Path $WorkerSource $ctx.WorkerExeName
foreach ($s in @(@($ServerSource, $stagedServerExe), @($WorkerSource, $stagedWorkerExe))) {
if (-not (Test-Path -LiteralPath $s[1])) { Fail "staged source '$($s[0])' does not contain $(Get-Leaf $s[1]) (the installed exe name; NSSM and $($script:EnvWorkerExe) point at it)" }
}
if ((Test-Inside $ServerSource $ctx.ServerDir) -or (Test-Inside $WorkerSource $ctx.WorkerDir) -or (Test-Inside $ServerSource $ctx.WorkerDir) -or (Test-Inside $WorkerSource $ctx.ServerDir)) { Fail 'staging directory is inside a live directory' }
$arch = Get-ExeArchitecture $stagedWorkerExe
if ($arch -ne 'X86') { Fail "staged Worker $stagedWorkerExe architecture is '$arch', not X86 (trap 5)" }
Info "staged Worker architecture: X86"
$sSha = Get-ProductSha $stagedServerExe
$wSha = Get-ProductSha $stagedWorkerExe
Info "staged Server SHA: $sSha ; staged Worker SHA: $wSha ; live Server SHA: $(Get-ProductSha $ctx.Application)"
if ($sSha -and $wSha -and -not (Test-ShaMatch $sSha $wSha)) { Fail "Server ($sSha) and Worker ($wSha) were built from different commits (wire protocol risk)" }
if ($ExpectedSha) {
if (-not (Test-ShaMatch $sSha $ExpectedSha)) {
if ($script:Dry) { Warn "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha' (dry-run: continuing)" }
else { Fail "staged Server SHA '$sSha' does not match -ExpectedSha '$ExpectedSha'" }
}
} else { Warn 'no -ExpectedSha given: the post-deploy SHA check will only record the SHA' }
$leafSha = if ((Get-Leaf $ctx.ServerDir) -match '([0-9a-f]{7,40})$') { $Matches[1] } else { $null }
if ($leafSha -and $sSha -and -not (Test-ShaMatch $leafSha $sSha)) { Warn "the Server dir name '$(Get-Leaf $ctx.ServerDir)' carries SHA $leafSha but will hold $sSha after the rename-in-place swap" }
# Runtime: never silently downgrade a self-contained Server.
$liveRt = Get-BundledRuntimeVersion $ctx.ServerDir; $newRt = Get-BundledRuntimeVersion $ServerSource
Info "bundled .NET runtime: live $(if ($liveRt) { $liveRt } else { 'none (framework-dependent)' }), staged $(if ($newRt) { $newRt } else { 'none (framework-dependent)' })"
if ($liveRt -and -not $newRt) { Warn 'live Server is self-contained, staged is framework-dependent: the box must have a matching .NET 10 runtime installed' }
if ($liveRt -and $newRt -and $newRt -lt $liveRt) {
if ($AllowRuntimeDowngrade) { Warn "staged runtime $newRt is OLDER than live $liveRt (-AllowRuntimeDowngrade)" }
else { Fail "staged self-contained runtime $newRt is OLDER than the live $liveRt; rebuild with a newer SDK or pass -AllowRuntimeDowngrade" }
}
# What the swap will carry over / drop (top-level files).
foreach ($pair in @(@('Server', $ctx.ServerDir, $ServerSource), @('Worker', $ctx.WorkerDir, $WorkerSource))) {
$plan = Get-CarryPlan $pair[1] $pair[2] ($pair[0] -eq 'Server' -and -not $TakeNewAppSettings)
Info "$($pair[0]): carry from live -> $(if ($plan.Carry.Count) { $plan.Carry -join ', ' } else { 'nothing' }); live-only binaries dropped -> $(if ($plan.Dropped.Count) { $plan.Dropped -join ', ' } else { 'none' })"
if ($plan.Dirs.Count) { Warn "$($pair[0]): live-only directories NOT carried (they stay in the .bak): $($plan.Dirs -join ', ')" }
}
$lw = @(Get-Files $ctx.WorkerDir | ForEach-Object { $_.Name }); $nw = @(Get-Files $WorkerSource | ForEach-Object { $_.Name })
$onlyNew = @($nw | Where-Object { $lw -notcontains $_ })
if ($onlyNew.Count) { Info "Worker: files only in the staged build: $($onlyNew -join ', ')" }
# Disk space: 2x (server + worker + stores + payload), at least MinFreeGB
if (-not $SkipFreeSpaceCheck) {
$need = (Get-DirSize $ctx.ServerDir) + (Get-DirSize $ctx.WorkerDir) + (Get-DirSize $ServerSource) + (Get-DirSize $WorkerSource)
foreach ($f in (Get-StoreFiles $ctx.AuthDb) + (Get-StoreFiles $ctx.SecretsDb)) { $need += (Get-Item -LiteralPath $f).Length }
$need = [math]::Max($need * 2, $MinFreeGB * 1GB)
$root = [IO.Path]::GetPathRoot((Resolve-Path -LiteralPath $ctx.ServerDir).Path)
$free = (New-Object IO.DriveInfo $root).AvailableFreeSpace
Info ("free on {0}: {1:N2} GB, required {2:N2} GB" -f $root, ($free / 1GB), ($need / 1GB))
if ($free -lt $need) { Fail 'not enough free disk space for the backups' }
}
Say '== Preflight OK =='
}
# --- stop / start ---------------------------------------------------------------------------------------------
function Wait-State([string]$Want, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { $s = Get-ServiceState; if ($s -eq $Want) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
function Wait-NoGatewayProcess($ctx, [int]$TimeoutSec) {
$deadline = (Get-Date).AddSeconds($TimeoutSec)
do { if (@(Get-GatewayProcesses $ctx).Count -eq 0) { return $true }; Start-Sleep -Seconds 1 } while ((Get-Date) -lt $deadline)
$false
}
# Kill only this install's gateway processes: path under the Server/Worker dir, or (path unreadable) the installed image name.
function Stop-InstallProcesses($ctx, [string]$Why) {
$installed = @((Get-Stem $ctx.Application), (Get-Stem $ctx.WorkerExe)) | Where-Object { $_ }
$mine = @(Get-GatewayProcesses $ctx | Where-Object {
if ($_.Path) { (Test-Inside $_.Path $ctx.ServerDir) -or (Test-Inside $_.Path $ctx.WorkerDir) } else { $installed -contains $_.Name }
} | Sort-Object { if ($_.Role -eq 'Server') { 0 } else { 1 } })
foreach ($p in $mine) { Say "[ACTION] Stop-Process -Id $($p.Id) -Force ($($p.Name), $Why)"; Stop-ProcessById $p.Id }
$mine.Count
}
function Stop-Gateway($ctx) {
Say '-- Stop --'
$state = Get-ServiceState
if ($state -eq 'SERVICE_STOPPED') { Info 'already stopped' }
else {
Step "nssm stop $ServiceName (wait up to $StopTimeoutSeconds s for SERVICE_STOPPED, then for every gateway/Worker process to exit)" {
[void](Invoke-Nssm -NssmArgs @('stop', $ServiceName) -AllowFail)
if (-not (Wait-State 'SERVICE_STOPPED' $StopTimeoutSeconds)) {
# STOP_PENDING trap: workers did not exit in time. Kill this install's Server, then its Workers.
Warn "service still $(Get-ServiceState) after $StopTimeoutSeconds s (STOP_PENDING trap) - killing this install's gateway processes"
$n = Stop-InstallProcesses $ctx 'STOP_PENDING'
if ($n -eq 0) { Warn 'no process of this install was found to kill' }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) {
Warn 'still not STOPPED; killing the NSSM service host process'
$svcPid = Get-ServiceHostPid
if ($svcPid) { Say "[ACTION] Stop-Process -Id $svcPid -Force (service host)"; Stop-ProcessById $svcPid }
if (-not (Wait-State 'SERVICE_STOPPED' 20)) { Fail "service '$ServiceName' will not stop; investigate manually (state $(Get-ServiceState))" }
}
}
if (-not (Wait-NoGatewayProcess $ctx 20)) {
Warn "gateway/Worker process(es) outlived the service stop (unclean stop): $(Format-Procs (Get-GatewayProcesses $ctx))"
[void](Stop-InstallProcesses $ctx 'left over after the stop')
[void](Wait-NoGatewayProcess $ctx 10)
}
}
}
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'after stop'
Info 'stopped; no gateway/Worker process on the box'
Assert-EnvUnchanged $ctx 'after the stop'
}
}
function Start-Gateway($ctx) {
Say '-- Start --'
if (-not $script:Dry) {
Assert-NoGatewayProcess $ctx 'before start (OrphanWorkerTerminator guard)'
Assert-EnvUnchanged $ctx 'before the start'
} else { Say "[DRYRUN] assert no process named $($script:ServerImages -join '|') / $($script:WorkerImages -join '|') exists, and the env block fingerprint is unchanged (OrphanWorkerTerminator guard)" }
Step "nssm start $ServiceName (once)" {
# nssm exits non-zero with 'Unexpected status SERVICE_START_PENDING' when the app is slow to report RUNNING;
# under ErrorActionPreference=Stop its stderr line throws. The RUNNING wait below is the real gate.
try { $null = & $NssmPath start $ServiceName 2>&1 } catch { Say "[info] nssm start: $($_.Exception.Message) -- deciding by the RUNNING wait below" }
}
Step "wait for SERVICE_RUNNING and compare the env block fingerprint" {
if (-not (Wait-State 'SERVICE_RUNNING' 60)) { Fail "service did not reach RUNNING (state $(Get-ServiceState))" }
Assert-EnvUnchanged $ctx 'after the start'
}
}
# --- backup / swap -----------------------------------------------------------------------------------------------
function Get-Suffix { if ($Label) { "$($script:Stamp)-$Label" } else { $script:Stamp } }
function Backup-State($ctx) {
Say '-- Backup (service stopped) --'
$suffix = Get-Suffix
$root = Split-Path -Parent $ctx.ServerDir
$dir = Join-Path $root "_deploy-backup-$suffix"
$script:BackupDir = $dir
Step "create $dir, restrict it to SYSTEM + Administrators, then save the env block (secrets), service registry key and env fingerprint" {
New-Item -ItemType Directory -Path $dir -Force | Out-Null
Protect-Directory $dir
$body = @("Application=$($ctx.Application)", "AppDirectory=$($ctx.AppDirectory)", '--- AppEnvironmentExtra ---') + @(Read-EnvBlockLines)
Set-Content -LiteralPath (Join-Path $dir 'AppEnvironmentExtra.txt') -Value $body -Encoding UTF8
Export-ServiceKey (Join-Path $dir "$ServiceName-servicekey.reg")
$fp = @("BlockHash=$($ctx.EnvFp.BlockHash)") + @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
Set-Content -LiteralPath (Join-Path $dir 'env-fingerprint.txt') -Value $fp -Encoding ASCII
}
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($f in (Get-StoreFiles $db)) {
$dst = "$f.bak-$suffix"
Step "copy store file $f -> $dst (live file is never moved)" {
Copy-Item -LiteralPath $f -Destination $dst -Force
if ((Get-FileSha $f) -ne (Get-FileSha $dst)) { Fail "store backup hash mismatch for $f" }
}
}
if (-not (Test-Path -LiteralPath $db)) { Info "no store at $db to back up" }
}
if ($ctx.KekFile -and (Test-Path -LiteralPath $ctx.KekFile)) {
Step "copy master key file $($ctx.KekFile) into $dir (live file is never moved)" { Copy-Item -LiteralPath $ctx.KekFile -Destination (Join-Path $dir (Get-Leaf $ctx.KekFile)) -Force }
}
}
function Test-Preserved([string]$Name) { foreach ($p in $script:PreservePatterns) { if ($Name -like $p) { return $true } }; $false }
# Which top-level files of the live dir go into the new one. Carry = names copied from live (live wins).
function Get-CarryPlan([string]$Live, [string]$Source, [bool]$KeepSettings) {
$liveFiles = @(Get-Files $Live)
$srcNames = @(Get-Files $Source | ForEach-Object { $_.Name })
$srcDirs = @(Get-Dirs $Source | ForEach-Object { $_.Name })
$carry = New-Object System.Collections.Generic.List[string]; $dropped = New-Object System.Collections.Generic.List[string]
foreach ($f in $liveFiles) {
$inSource = $srcNames -contains $f.Name
if ($f.Name -like 'appsettings*.json') { if ($KeepSettings) { $carry.Add($f.Name) }; continue }
if (Test-Preserved $f.Name) { $carry.Add($f.Name); continue }
if ($inSource) { continue }
if ($script:BinaryExtensions -contains $f.Extension.ToLowerInvariant()) { $dropped.Add($f.Name) } else { $carry.Add($f.Name) }
}
$dirs = @(Get-Dirs $Live | Where-Object { $srcDirs -notcontains $_.Name } | ForEach-Object { $_.Name })
[pscustomobject]@{ Carry = $carry.ToArray(); Dropped = $dropped.ToArray(); Dirs = $dirs }
}
function Swap-Directory([string]$Live, [string]$Source, [string]$Name, [bool]$KeepSettings) {
$bak = "$Live.bak-$(Get-Suffix)"
$plan = Get-CarryPlan $Live $Source $KeepSettings
$hashes = @{}
foreach ($f in $plan.Carry) { $hashes[$f] = Get-FileSha (Join-Path $Live $f) }
$sddlOld = Get-DirSddl $Live
Step "rename $Name dir: $Live -> $bak" { Rename-Item -LiteralPath $Live -NewName (Split-Path -Leaf $bak) }
Step "copy new ${Name}: $Source -> $Live" { Copy-Item -LiteralPath $Source -Destination $Live -Recurse -Force }
foreach ($f in $plan.Carry) {
Step "carry live $Name\$f forward from $bak (sha256 $($hashes[$f].Substring(0, 12)))" {
Copy-Item -LiteralPath (Join-Path $bak $f) -Destination (Join-Path $Live $f) -Force
if ((Get-FileSha (Join-Path $Live $f)) -ne $hashes[$f]) { Fail "carried $Name\$f does not match the live copy" }
Copy-ExplicitAcl (Join-Path $bak $f) (Join-Path $Live $f)
}
}
foreach ($f in $plan.Dropped) { Info "$Name\$f exists only in the old build (binary) and is not carried; it stays in $bak" }
if (-not $script:Dry) {
$sddlNew = Get-DirSddl $Live
if ($sddlOld -and $sddlNew -and $sddlOld -ne $sddlNew) { Warn "$Name dir ACL differs from the old one (new inherits from the parent). old: $sddlOld new: $sddlNew" }
}
$bak
}
# --- verification ----------------------------------------------------------------------------------------------------
function Invoke-Verify($ctx, [bool]$AfterRollback = $false) {
Say '== Verify =='
if ($script:Dry) {
Say "[DRYRUN] GET $HealthBaseUrl/health/ready until 200 (<= $ReadyTimeoutSeconds s)"
Say "[DRYRUN] GET $HealthBaseUrl/healthz ; read ProductVersion SHA of $($ctx.Application)$(if ($ExpectedSha -and -not $Rollback) { " and require it to match $ExpectedSha" })"
Say "[DRYRUN] exactly one gateway Server ($($ctx.Application)) on the box, same PID, service SERVICE_RUNNING for $StableSeconds s, env fingerprint unchanged"
return
}
$ready = $false; $deadline = (Get-Date).AddSeconds($ReadyTimeoutSeconds)
while (-not $ready -and (Get-Date) -lt $deadline) {
$ready = ((Get-HttpStatus "$HealthBaseUrl/health/ready") -eq 200)
if (-not $ready) { Start-Sleep -Seconds 3 }
}
if (-not $ready) { Fail "$HealthBaseUrl/health/ready did not return 200 within $ReadyTimeoutSeconds s" }
Info '/health/ready = 200'
$hz = Get-HttpStatus "$HealthBaseUrl/healthz"
if ($hz -ne 200) { Fail "/healthz = $hz" }
Info '/healthz = 200'
$sha = Get-ProductSha $ctx.Application
Info "deployed Server SHA (ProductVersion): $sha"
if ($ExpectedSha -and -not $Rollback -and -not $AfterRollback -and -not (Test-ShaMatch $sha $ExpectedSha)) { Fail "deployed SHA '$sha' != expected '$ExpectedSha'" }
Assert-SingleGateway $ctx 'SERVICE_RUNNING'
$pid0 = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })[0].Id
Info "watching the service for $StableSeconds s (gateway #$pid0)"
$until = (Get-Date).AddSeconds($StableSeconds)
do {
if ((Get-ServiceState) -ne 'SERVICE_RUNNING') { Fail 'service left RUNNING during the stability window (restart loop?)' }
Start-Sleep -Seconds 5
} while ((Get-Date) -lt $until)
$servers = @(Get-GatewayProcesses $ctx | Where-Object { $_.Role -eq 'Server' })
if ($servers.Count -ne 1 -or $servers[0].Id -ne $pid0) { Fail "gateway process changed during the stability window: #$pid0 -> $(Format-Procs $servers) (restart loop or second gateway)" }
Assert-EnvUnchanged $ctx 'after the stability window'
Warn 'health is green with zero sessions; still prove the Worker path: open a session with a PRE-EXISTING API key and read a known tag (runbook 7.3)'
}
# --- rollback ------------------------------------------------------------------------------------------------------------
# Backup dirs come in two spellings: <leaf>.bak-yyyyMMdd-HHmmss[-label] (this script) and <leaf>.bak.yyyyMMddTHHmmss (manual / adapted scripts).
function Get-BackupKey([string]$Name) {
if ($Name -match '\.bak[-.](\d{8})[-T]?(\d{6})') { return "$($Matches[1])$($Matches[2])" }
$null
}
function Find-Backup([string]$Live, [string]$Stamp) {
$leaf = Split-Path -Leaf $Live; $parent = Split-Path -Parent $Live
$cands = @(Get-Dirs $parent | Where-Object { ($_.Name -like "$leaf.bak-*" -or $_.Name -like "$leaf.bak.*") -and (Get-BackupKey $_.Name) })
if ($Stamp) {
$key = ($Stamp -replace '[^0-9]', '')
$cands = @($cands | Where-Object { (Get-BackupKey $_.Name) -eq $key -or $_.Name -like "$leaf.bak-$Stamp*" -or $_.Name -like "$leaf.bak.$Stamp*" })
}
$cands | Sort-Object { Get-BackupKey $_.Name } -Descending | Select-Object -First 1
}
function Invoke-RollbackSwap($ctx, [string]$Stamp, [bool]$Stores, [bool]$AllowPartial) {
Say '== Rollback =='
$sb = Find-Backup $ctx.ServerDir $Stamp
if (-not $sb -and -not $AllowPartial) { Fail "no Server backup directory found (stamp '$Stamp')" }
$key = if ($sb) { Get-BackupKey $sb.Name } else { ($Stamp -replace '[^0-9]', '') }
$wb = Find-Backup $ctx.WorkerDir $key
if ($wb -and (Get-BackupKey $wb.Name) -ne $key) { $wb = $null }
if (-not $wb -and -not $AllowPartial) { Fail "no Worker backup matching the Server backup $($sb.Name) (stamp key $key); refusing a mixed rollback" }
if (-not $sb -and -not $wb) { Fail "nothing to roll back to (stamp '$Stamp')" }
Info "rolling back to Server=$(if ($sb) { $sb.Name } else { '(unchanged)' }) Worker=$(if ($wb) { $wb.Name } else { '(unchanged)' })"
$fpFile = @(Get-Dirs (Split-Path -Parent $ctx.ServerDir) '_deploy-backup-*' | Where-Object { ($_.Name -replace '[^0-9]', '').StartsWith($key) } | ForEach-Object { Join-Path $_.FullName 'env-fingerprint.txt' } | Where-Object { Test-Path -LiteralPath $_ })
if ($fpFile.Count) {
$saved = Get-Content -LiteralPath $fpFile[0] | Where-Object { $_ -notlike 'BlockHash=*' }
$now = @($ctx.EnvFp.Entries.Keys | ForEach-Object { "$_=$($ctx.EnvFp.Entries[$_])" })
if (($saved -join '|') -ne ($now -join '|')) { Warn "the env block differs from the one saved before the upgrade ($($fpFile[0])); restore it from AppEnvironmentExtra.txt there if that was not intended (runbook section 8 step 4)" }
else { Info "env block matches the pre-upgrade fingerprint in $($fpFile[0])" }
}
Stop-Gateway $ctx
foreach ($pair in @(@($ctx.ServerDir, $sb), @($ctx.WorkerDir, $wb))) {
if (-not $pair[1]) { Info "$($pair[0]) has no backup for this stamp; left unchanged"; continue }
$failed = "$($pair[0]).failed-$($script:Stamp)"
if (Test-Path -LiteralPath $pair[0]) { Step "rename $($pair[0]) -> $failed" { Rename-Item -LiteralPath $pair[0] -NewName (Split-Path -Leaf $failed) } }
Step "rename $($pair[1].FullName) -> $($pair[0])" { Rename-Item -LiteralPath $pair[1].FullName -NewName (Split-Path -Leaf $pair[0]) }
}
if ($Stores) {
$bstamp = ($sb.Name -replace '^.*\.bak[-.]', '')
foreach ($db in @($ctx.AuthDb, $ctx.SecretsDb)) {
foreach ($ext in '', '-wal', '-shm') {
$live = "$db$ext"; $bak = "$db$ext.bak-$bstamp"
if (Test-Path -LiteralPath $bak) { Step "restore store file $bak -> $live" { Copy-Item -LiteralPath $bak -Destination $live -Force } }
elseif ($ext -and (Test-Path -LiteralPath $live) -and (Test-Path -LiteralPath "$db.bak-$bstamp")) {
# a stale wal/shm must not pair with the restored db: move it aside, never delete
Step "move stale $live aside -> $live.stale-$($script:Stamp)" { Rename-Item -LiteralPath $live -NewName ("$(Split-Path -Leaf $live).stale-$($script:Stamp)") }
}
elseif (-not $ext) { Warn "no backup $bak for $db; live store left untouched" }
}
}
} else { Info 'stores left untouched (they live outside the swap); pass -RestoreStores after a schema bump' }
Start-Gateway $ctx
}
# --- main -------------------------------------------------------------------------------------------------------------------
function Invoke-Main {
$exit = 0
$ctx = $null
try {
Say "Upgrade-MxGatewayService stamp=$($script:Stamp) mode=$(if ($Rollback) {'ROLLBACK'} else {'UPGRADE'}) dry=$($script:Dry) host=$env:COMPUTERNAME"
$ctx = Get-Context
Invoke-Preflight $ctx
if (-not $script:Dry) {
$lockPath = Join-Path (Split-Path -Parent $ctx.ServerDir) '.mxgw-deploy.lock'
try { $script:LockStream = [IO.File]::Open($lockPath, 'OpenOrCreate', 'ReadWrite', 'None') }
catch { Fail "another deploy run holds $lockPath" }
}
if ($Rollback) {
Invoke-RollbackSwap $ctx $BackupStamp $RestoreStores.IsPresent $false
Invoke-Verify $ctx
} else {
Stop-Gateway $ctx
try { Backup-State $ctx }
catch {
$err = $_
if (-not $script:Dry) {
Warn 'backup failed before anything was swapped - starting the unchanged build again'
try { Start-Gateway $ctx } catch { Warn "restart of the unchanged build FAILED: $($_.Exception.Message)" }
}
throw $err
}
$touched = $false
try {
$touched = $true
$null = Swap-Directory $ctx.ServerDir $ServerSource 'Server' (-not $TakeNewAppSettings)
$null = Swap-Directory $ctx.WorkerDir $WorkerSource 'Worker' $false
if (-not $script:Dry) { Assert-EnvUnchanged $ctx 'after the swap' }
Start-Gateway $ctx
Invoke-Verify $ctx
} catch {
$err = $_
Say "[FAIL] $(Hide $err.Exception.Message)"
if ($script:Dry -or $NoAutoRollback -or -not $touched) { throw }
Warn 'upgrade failed - rolling back automatically'
$script:EnvWarnOnly = $true
try {
Invoke-RollbackSwap $ctx $script:Stamp $false $true
Invoke-Verify $ctx $true
Warn 'ROLLED BACK to the previous build; original failure follows'
} catch { Warn "AUTOMATIC ROLLBACK FAILED: $($_.Exception.Message). Manual intervention required (backups: *.bak-$(Get-Suffix))." }
throw $err
}
}
Say '== DONE =='
} catch {
$exit = 1
$m = Hide $_.Exception.Message
$script:Notes.Add("ERROR: $m")
Write-Host "ERROR: $m" -ForegroundColor Red
} finally {
if ($script:LockStream) { $script:LockStream.Dispose(); $script:LockStream = $null }
try {
if (-not $script:Dry -and $ctx) {
$notes = Join-Path (Split-Path -Parent $ctx.ServerDir) "_DEPLOY-NOTES-$($script:Stamp).txt"
Set-Content -LiteralPath $notes -Value $script:Notes -Encoding UTF8
Write-Host "deploy notes: $notes"
}
} catch { }
$script:Redact.Clear()
}
$exit
}
# Dot-sourcing (the dry-run harness) only defines the functions.
if ($MyInvocation.InvocationName -ne '.') { $rc = Invoke-Main | Select-Object -Last 1; exit $rc }
@@ -0,0 +1,117 @@
{
"Serilog": {
"Using": [
"Serilog.Sinks.Console",
"Serilog.Sinks.File"
],
"MinimumLevel": {
"Default": "Information",
"Override": {
"Microsoft.AspNetCore": "Warning"
}
},
"WriteTo": [
{
"Name": "Console"
},
{
"Name": "File",
"Args": {
"path": "logs/mxgateway-.log",
"rollingInterval": "Day",
"retainedFileCountLimit": 30,
"fileSizeLimitBytes": 52428800,
"rollOnFileSizeLimit": true
}
}
]
},
"AllowedHosts": "*",
"Secrets": {
"MasterKey": {
"Source": "Environment",
"EnvVarName": "ZB_SECRETS_MASTER_KEY"
},
"RunMigrationsOnStartup": true,
"ResolveCacheTtl": "00:00:30"
},
"MxGateway": {
"Authentication": {
"Mode": "ApiKey",
"PepperSecretName": "MxGateway:ApiKeyPepper",
"RunMigrationsOnStartup": true
},
"Ldap": {
"Enabled": true,
"Server": "localhost",
"Port": 3893,
"Transport": "None",
"AllowInsecure": true,
"SearchBase": "dc=zb,dc=local",
"ServiceAccountDn": "cn=serviceaccount,dc=zb,dc=local",
"ServiceAccountPassword": "${secret:ldap/mxgateway/bind}",
"UserNameAttribute": "cn",
"DisplayNameAttribute": "cn",
"GroupAttribute": "memberOf"
},
"Worker": {
"ExecutablePath": "src\\ZB.MOM.WW.MxGateway.Worker\\bin\\x86\\Release\\ZB.MOM.WW.MxGateway.Worker.exe",
"RequiredArchitecture": "X86",
"StartupTimeoutSeconds": 30,
"ShutdownTimeoutSeconds": 10,
"HeartbeatCheckIntervalSeconds": 5,
"HeartbeatGraceSeconds": 15,
"HeartbeatStuckCeilingSeconds": 75,
"WorkerHeartbeatSendIntervalSeconds": 5,
"WorkerHeartbeatGraceSeconds": 15,
"WorkerHeartbeatStuckCeilingSeconds": 75,
"EventChannelFullModeTimeoutMilliseconds": 5000,
"MaxMessageBytes": 16842752
},
"Sessions": {
"DefaultCommandTimeoutSeconds": 30,
"MaxSessions": 64,
"MaxPendingCommandsPerSession": 128,
"DefaultLeaseSeconds": 1800,
"LeaseSweepIntervalSeconds": 30,
"DetachGraceSeconds": 30,
"UnattachedSessionGraceSeconds": 0,
"AllowMultipleEventSubscribers": false,
"MaxEventSubscribersPerSession": 8
},
"Events": {
"QueueCapacity": 10000,
"BackpressurePolicy": "FailFast",
"ReplayBufferCapacity": 1024,
"ReplayRetentionSeconds": 300,
"ReplayBufferMaxBytes": 67108864
},
"Dashboard": {
"Enabled": true,
"AllowAnonymousLocalhost": true,
"SnapshotIntervalMilliseconds": 1000,
"RecentFaultLimit": 100,
"RecentSessionLimit": 200,
"ShowTagValues": false
},
"Protocol": {
"WorkerProtocolVersion": 1,
"MaxGrpcMessageBytes": 16777216
},
"Galaxy": {
"ConnectionString": "Server=localhost;Database=ZB;Integrated Security=True;TrustServerCertificate=True;Encrypt=False;",
"CommandTimeoutSeconds": 60,
"DashboardRefreshIntervalSeconds": 30,
"PersistSnapshot": true
},
"Alarms": {
"Enabled": true,
"SubscriptionExpression": "\\\\DESKTOP-6JL3KKO\\Galaxy!DEV",
"DefaultArea": "",
"ReconcileIntervalSeconds": 30,
"PollIntervalMilliseconds": 500,
"MaxAlarmsPerFetch": 1024,
"AckClientName": "MxGateway"
}
}
}
@@ -0,0 +1,99 @@
# PROD MxAccessGateway first install (plan Phase 4 + S3A). Run ON the box, only after the owner's go for Phase 4.
# -Mode dry : verify the zip, the MXAccess gate and the env, expand into .\dry\, run Install-MxGatewayService -DryRun.
# Creates nothing outside this stage folder.
# -Mode real : expand into E:\ApiInstall\MxAccessGateway\{app,worker}, put appsettings.json.prod in place, create the
# KEK (or take -KekFrom), install the NSSM service STOPPED (never -Start: seed the LDAP secret first, S3A.4).
# -ValuesDir holds one <KEY>.txt per '@file' line in mxgw.env (pepper, Galaxy connection string). It is read and never
# printed; the caller deletes it afterwards. Secret values never go on argv.
param(
[Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode,
[string]$ValuesDir,
[string]$KekFrom
)
$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'
$stage = $PSScriptRoot
$zip = Join-Path $stage 'mxgw-e74918cc.zip'
$zipSha = 'F0999B75BEE196375AEB8E3E277A0E948C54A747FD5251D336461C9665D847F2'
$root = 'E:\ApiInstall\MxAccessGateway'
$kekDir = 'C:\ProgramData\ZB\kek'
$kek = "$kekDir\mxgateway.kek"
$nssm = 'E:\psrem-stage\prod-common\nssm.exe'
$lmx = 'C:\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
function Fail($m) { throw "ABORT: $m" }
Say "install-mxgw mode=$Mode box=$env:COMPUTERNAME"
# 1. MXAccess gate (plan Q3): the PAR boxes have none until WinPlatform is deployed from the prod Galaxy.
if (-not (Test-Path -LiteralPath $lmx)) { Fail "no $lmx - MXAccess not installed on this box (plan Q3 gate)" }
Say "LmxProxy.dll $((Get-Item $lmx).VersionInfo.FileVersion) sha256 $((Get-FileHash $lmx).Hash.Substring(0,12))"
# 2. artifact
$h = (Get-FileHash $zip -Algorithm SHA256).Hash
if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" }
Say 'zip sha256 ok'
if (-not (Test-Path $nssm)) { Fail "nssm not staged at $nssm" }
# 3. env: placeholders + @file values
$lines = Get-Content (Join-Path $stage 'mxgw.env')
$left = @($lines | Where-Object { $_ -notmatch '^\s*#' -and $_ -match '__Q\d+_' })
if ($left.Count) { Fail "unresolved placeholders in mxgw.env: $(($left | ForEach-Object { ($_ -split '=')[0] }) -join ', ')" }
$out = New-Object System.Collections.Generic.List[string]
foreach ($l in $lines) {
if ($l -match '^\s*(#|$)') { continue }
$k, $v = $l -split '=', 2
if ($v -eq '@file') {
if (-not $ValuesDir) { Fail "$k needs -ValuesDir" }
$f = Join-Path $ValuesDir "$k.txt"
if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" }
$v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n")
if (-not $v) { Fail "empty value file for $k" }
}
$out.Add("$k=$v")
}
Say "env entries: $($out.Count)"
if ($out | Where-Object { $_ -match '_(DEV|QA)=' }) { Fail 'a _DEV/_QA group name is in a PROD env (plan S3A.2)' }
# 4. layout
if ($Mode -eq 'dry') { $base = Join-Path $stage 'dry' } else { $base = $root }
$app = Join-Path $base 'app'; $worker = Join-Path $base 'worker'
if ($Mode -eq 'real') {
if ((Test-Path $app) -or (Test-Path $worker)) { Fail "$app or $worker already exists - this is the first-install script; use Upgrade-MxGatewayService.ps1" }
if (Get-Service MxAccessGw -ErrorAction SilentlyContinue) { Fail 'service MxAccessGw already exists' }
} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force }
$tmp = Join-Path $stage "expand-$(Get-Date -Format yyyyMMddTHHmmss)"
Expand-Archive -LiteralPath $zip -DestinationPath $tmp
New-Item -ItemType Directory -Force $base | Out-Null
Move-Item (Join-Path $tmp 'Server') $app
Move-Item (Join-Path $tmp 'Worker') $worker
Remove-Item $tmp -Recurse -Force
Copy-Item (Join-Path $stage 'appsettings.json.prod') (Join-Path $app 'appsettings.json') -Force
if (Select-String -Path (Join-Path $app 'appsettings.json') -Pattern 'GwAdmin|GwReader' -Quiet) { Fail 'GwAdmin/GwReader still in appsettings.json' }
Say "expanded to $base (app: $(@(Get-ChildItem $app -Recurse -File).Count) files, worker: $(@(Get-ChildItem $worker -Recurse -File).Count) files)"
foreach ($e in (Join-Path $app 'ZB.MOM.WW.MxGateway.Server.exe'), (Join-Path $worker 'MxGateway.Worker.exe')) {
Say "$(Split-Path $e -Leaf) $((Get-Item $e).VersionInfo.ProductVersion)" }
# 5. KEK (32 raw random bytes; Administrators + SYSTEM only)
if ($Mode -eq 'real') {
New-Item -ItemType Directory -Force $kekDir | Out-Null
& icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null
if (Test-Path $kek) { Say "KEK exists, kept: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" }
elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say "KEK copied from -KekFrom: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))" }
else {
$b = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b)
[IO.File]::WriteAllBytes($kek, $b); Say "KEK generated: sha256 $((Get-FileHash $kek).Hash.Substring(0,8))"
}
if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' }
}
# 6. service (stopped). Env goes through a protected temp file that is always deleted.
$envTmp = Join-Path $stage "_env-$PID.tmp"
try {
[IO.File]::WriteAllLines($envTmp, $out)
& icacls.exe $envTmp /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
$a = @{ ServerDir = $app; WorkerDir = $worker; NssmPath = $nssm; DataDir = "$root\data"; LogDir = "$root\logs"
EnvFile = $envTmp; WorkerExeName = 'MxGateway.Worker.exe' }
if ($Mode -eq 'dry') { $a.DryRun = $true }
& (Join-Path $stage 'Install-MxGatewayService.ps1') @a
$rc = $LASTEXITCODE
} finally { Remove-Item $envTmp -Force -ErrorAction SilentlyContinue }
if ($rc -ne 0) { Fail "Install-MxGatewayService exited $rc" }
if ($Mode -eq 'real') { Say 'INSTALLED STOPPED. Next: seed nam-ldap-svc-password (prod-common\seed-secret-prod.ps1), review the env, then start (plan S3A.4/S3A.5).' }
Say "DONE mode=$Mode"
@@ -0,0 +1,42 @@
# MxAccessGateway NSSM env for PROD (all four boxes; QA vq03 key set + PROD plan §3A.3). KEY=VALUE, '#' comments.
# @file = value read at install time from <ValuesDir>\<KEY>.txt (secrets; never in this file)
# __Q2_..__ = open question Q2 (prod Galaxy Repository / alarm area); install-mxgw.ps1 refuses while any remain
ASPNETCORE_ENVIRONMENT=Production
DOTNET_ENVIRONMENT=Production
Kestrel__Endpoints__Grpc__Url=http://0.0.0.0:5120
Kestrel__Endpoints__Grpc__Protocols=Http2
Kestrel__Endpoints__Dashboard__Url=http://0.0.0.0:5130
Kestrel__Endpoints__Dashboard__Protocols=Http1
MxGateway__Worker__ExecutablePath=E:\ApiInstall\MxAccessGateway\worker\MxGateway.Worker.exe
MxGateway__Worker__RequiredArchitecture=X86
MxGateway__Worker__WorkingDirectory=E:\ApiInstall\MxAccessGateway\worker
MxGateway__Worker__MaxMessageBytes=16842752
MxGateway__Protocol__MaxGrpcMessageBytes=16777216
MxGateway__Galaxy__ConnectionString=@file
MxGateway__Authentication__Mode=ApiKey
MxGateway__Authentication__SqlitePath=E:\ApiInstall\MxAccessGateway\data\gateway-auth.db
MxGateway__ApiKeyPepper=@file
MxGateway__Alarms__Enabled=true
MxGateway__Alarms__SubscriptionExpression=__Q2_ALARM_EXPRESSION__
MxGateway__Ldap__Enabled=true
MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com
MxGateway__Ldap__Port=636
MxGateway__Ldap__Transport=Ldaps
MxGateway__Ldap__AllowInsecure=false
MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636
MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com
MxGateway__Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com
MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}
MxGateway__Ldap__UserNameAttribute=sAMAccountName
MxGateway__Ldap__DisplayNameAttribute=name
MxGateway__Ldap__GroupAttribute=memberOf
MxGateway__Ldap__BindMode=ServiceAccount
MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_PROD=Administrator
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_PROD=Viewer
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_PROD=Viewer
MxGateway__Dashboard__RequireHttpsCookie=false
Secrets__SqlitePath=E:\ApiInstall\MxAccessGateway\data\secrets.db
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\mxgateway.kek
MxGateway__Telemetry__Exporter=Otlp
MxGateway__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
@@ -0,0 +1,116 @@
<#
.SYNOPSIS
Installs (or removes) the ZbOtelCollector service on a wonder box (scadaproj#54).
.DESCRIPTION
Layout: E:\ApiInstall\OtelCollector\{bin\otelcol-contrib.exe, bin\nssm.exe, config\config.yaml, data\, logs\}
Service: ZbOtelCollector (NSSM), virtual account NT SERVICE\ZbOtelCollector, Automatic, restart on exit.
ACL: the root is protected (inherited ACEs copied, so an inherited OtOpcUaScriptWorker deny stays as an explicit
ACE), BUILTIN\Users removed; SYSTEM + Administrators full; the service account gets RX on the tree and
Modify on data\ and logs\.
Env: ZB_TIER (DEV|QA, stamped as deployment.environment.name) and NO_WINDOWS_SERVICE=1 (NSSM is the service
wrapper, so otelcol runs in console mode).
Every listener in config.yaml binds 127.0.0.1. The script checks that after start.
.PARAMETER StageDir Folder holding otelcol-contrib_0.162.0_windows_amd64.zip, config.yaml and SHA256SUMS.txt.
.PARAMETER Tier DEV, QA or PROD (PROD added in the 2026-11-19 PROD stage copy).
.PARAMETER NssmPath An existing nssm.exe on the box, copied into bin\.
.PARAMETER Uninstall Stop and remove the service. The folder is renamed to OtelCollector.removed.<ts> (kept).
#>
param(
[Parameter(Mandatory)] [string]$StageDir,
[ValidateSet('DEV','QA','PROD')] [string]$Tier,
[string]$NssmPath,
[switch]$Uninstall
)
$ErrorActionPreference = 'Stop'
$svcName = 'ZbOtelCollector'
$root = 'E:\ApiInstall\OtelCollector'
$acct = "NT SERVICE\$svcName"
$exeSha = 'F8F69BD67025F29353B44440069046CA5BDB50CBBFCD6F359D465F0FE84E436E'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
if ($Uninstall) {
$s = Get-Service $svcName -ErrorAction SilentlyContinue
if ($s) {
if ($s.Status -ne 'Stopped') { Stop-Service $svcName -Force; (Get-Service $svcName).WaitForStatus('Stopped', [TimeSpan]::FromSeconds(60)) }
& sc.exe delete $svcName | Out-Null
Say "service $svcName deleted"
}
if (Test-Path $root) { $dst = "$root.removed.$(Get-Date -Format yyyyMMddTHHmmss)"; Rename-Item $root $dst; Say "folder kept as $dst" }
return
}
if (-not $Tier -or -not $NssmPath) { throw '-Tier and -NssmPath are required for install' }
if (Get-Service $svcName -ErrorAction SilentlyContinue) { throw "$svcName already exists" }
if (Test-Path $root) { throw "$root already exists" }
# 1. Verify the staged files against SHA256SUMS.txt (written on the Mac from the release .sha256 + git).
$sums = @{}
Get-Content (Join-Path $StageDir 'SHA256SUMS.txt') | Where-Object { $_ -match '^\s*([0-9a-fA-F]{64})\s+\*?(\S+)\s*$' } |
ForEach-Object { $sums[$Matches[2]] = $Matches[1].ToUpperInvariant() }
foreach ($f in 'otelcol-contrib_0.162.0_windows_amd64.zip','config.yaml') {
$h = (Get-FileHash (Join-Path $StageDir $f) -Algorithm SHA256).Hash
if ($h -ne $sums[$f]) { throw "hash mismatch $f" }
Say "hash ok $f $($h.Substring(0,12))"
}
# 2. Layout
foreach ($d in 'bin','config','data','logs') { New-Item -ItemType Directory -Force -Path (Join-Path $root $d) | Out-Null }
$tmp = Join-Path $StageDir 'x'
if (Test-Path $tmp) { Remove-Item $tmp -Recurse -Force }
Expand-Archive (Join-Path $StageDir 'otelcol-contrib_0.162.0_windows_amd64.zip') -DestinationPath $tmp
$h = (Get-FileHash (Join-Path $tmp 'otelcol-contrib.exe') -Algorithm SHA256).Hash
if ($h -ne $exeSha) { throw "exe hash mismatch $h" }
Move-Item (Join-Path $tmp 'otelcol-contrib.exe') (Join-Path $root 'bin\otelcol-contrib.exe')
Remove-Item $tmp -Recurse -Force
Copy-Item $NssmPath (Join-Path $root 'bin\nssm.exe')
Copy-Item (Join-Path $StageDir 'config.yaml') (Join-Path $root 'config\config.yaml')
Say "layout ok, exe sha $($exeSha.Substring(0,12))"
# 3. Service
$nssm = Join-Path $root 'bin\nssm.exe'
& $nssm install $svcName (Join-Path $root 'bin\otelcol-contrib.exe') | Out-Null
$set = @(
@('AppParameters', "--config=$root\config\config.yaml"),
@('AppDirectory', $root),
@('DisplayName', 'ZB OpenTelemetry Collector'),
@('Description', 'otelcol-contrib 0.162.0, OTLP on 127.0.0.1:4317/4318 for the ZB.MOM.WW apps on this box (scadaproj#54)'),
@('Start', 'SERVICE_AUTO_START'),
@('AppStdout', "$root\logs\collector.log"),
@('AppStderr', "$root\logs\collector.log"),
@('AppRotateFiles', '1'), @('AppRotateOnline', '1'), @('AppRotateBytes', '10485760'),
@('AppExit', 'Default', 'Restart'), @('AppRestartDelay', '5000'))
foreach ($a in $set) { & $nssm set $svcName @a | Out-Null; if ($LASTEXITCODE -ne 0) { throw "nssm set $($a[0]) failed" } }
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svcName\Parameters" -Name AppEnvironmentExtra -Type MultiString -Value @("ZB_TIER=$Tier", 'NO_WINDOWS_SERVICE=1')
& sc.exe config $svcName obj= $acct | Out-Null
if ($LASTEXITCODE -ne 0) { throw "sc config obj failed $LASTEXITCODE" }
Say "service created, account $((Get-CimInstance Win32_Service -Filter "Name='$svcName'").StartName)"
# 4. ACL
& icacls $root /inheritance:d | Out-Null
& icacls $root /remove:g 'BUILTIN\Users' /T /C /Q | Out-Null
& icacls $root /grant "${acct}:(OI)(CI)(RX)" /Q | Out-Null
foreach ($d in 'data','logs') { & icacls (Join-Path $root $d) /grant "${acct}:(OI)(CI)(M)" /Q | Out-Null }
(icacls $root) | Where-Object { $_.Trim() } | ForEach-Object { " acl " + $_.Trim() }
# 5. Start + checks
Start-Service $svcName
$ok = $false
for ($i = 0; $i -lt 30; $i++) {
Start-Sleep 2
try { $r = Invoke-WebRequest -UseBasicParsing -TimeoutSec 3 'http://127.0.0.1:13133/'; if ($r.StatusCode -eq 200) { $ok = $true; break } } catch {}
}
Say "service $((Get-Service $svcName).Status); health_check ok=$ok"
if (-not $ok) { Get-Content "$root\logs\collector.log" -Tail 30; throw 'collector not healthy' }
$procId = (Get-CimInstance Win32_Service -Filter "Name='$svcName'").ProcessId
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$procId" | Where-Object Name -eq 'otelcol-contrib.exe'
$listen = Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId
$listen | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
$bad = $listen | Where-Object { $_.LocalAddress -ne '127.0.0.1' }
if ($bad) { throw "non-loopback listener: $($bad | ForEach-Object { "$($_.LocalAddress):$($_.LocalPort)" })" }
$r = Invoke-WebRequest -UseBasicParsing -Method Post -ContentType 'application/json' -Body '{}' 'http://127.0.0.1:4318/v1/traces'
Say "OTLP/HTTP empty POST -> $($r.StatusCode)"
$err = Select-String -Path "$root\logs\collector.log" -Pattern '"level":"error"|\terror\t' -SimpleMatch:$false
Say "collector.log error lines: $(@($err).Count)"
Get-Content "$root\logs\collector.log" | Select-String 'Everything is ready|Starting otelcol|Starting extensions' | ForEach-Object { " log " + $_.Line.Substring(0, [Math]::Min(160, $_.Line.Length)) }
@@ -0,0 +1,65 @@
<#
.SYNOPSIS
Read-only verification of ZbOtelCollector on this box (scadaproj#54): health, listeners, what each app sent.
.PARAMETER SinceMinutes Only count export requests whose newest timestamp is within this window.
.PARAMETER AppPorts App HTTP ports whose /metrics must still answer 200 (Prometheus stays served under OTLP).
#>
param([int]$SinceMinutes = 15, [int[]]$AppPorts = @(), [int]$Tail = 20)
$ErrorActionPreference = 'Continue'
$root = 'E:\ApiInstall\OtelCollector'
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
$s = Get-CimInstance Win32_Service -Filter "Name='ZbOtelCollector'"
Say "service $($s.State) $($s.StartMode) $($s.StartName)"
try { Say ("health_check " + (Invoke-WebRequest -UseBasicParsing -TimeoutSec 5 'http://127.0.0.1:13133/').StatusCode) } catch { Say "health_check FAIL $($_.Exception.Message)" }
$child = Get-CimInstance Win32_Process -Filter "ParentProcessId=$($s.ProcessId)" | Where-Object Name -eq 'otelcol-contrib.exe'
if ($child) { Get-NetTCPConnection -State Listen -OwningProcess $child.ProcessId | Sort-Object LocalPort | ForEach-Object { " listen $($_.LocalAddress):$($_.LocalPort)" }
" collector WS MB " + [int]((Get-Process -Id $child.ProcessId).WorkingSet64/1MB) }
Get-ChildItem "$root\data" | ForEach-Object { " file {0} {1:N1} MB {2:HH:mm:ss}" -f $_.Name, ($_.Length/1MB), $_.LastWriteTime }
" E: free GB " + [int]((Get-PSDrive E).Free/1GB)
$cut = [DateTimeOffset]::UtcNow.AddMinutes(-$SinceMinutes).ToUnixTimeMilliseconds() * 1000000
function Attr($res, $k) { ($res.attributes | Where-Object key -eq $k | Select-Object -First 1).value.stringValue }
foreach ($sig in 'metrics','traces','logs') {
$f = "$root\data\$sig.jsonl"
if (-not (Test-Path $f)) { Say "${sig}: no file"; continue }
$agg = @{}
foreach ($line in (Get-Content $f -Tail $Tail)) {
try { $o = $line | ConvertFrom-Json } catch { continue }
$rs = if ($sig -eq 'metrics') { $o.resourceMetrics } elseif ($sig -eq 'traces') { $o.resourceSpans } else { $o.resourceLogs }
foreach ($r in $rs) {
$key = "{0} | {1} | inst={2} | env={3} | site={4} role={5}" -f (Attr $r.resource 'service.name'), (Attr $r.resource 'host.name'), (Attr $r.resource 'service.instance.id'), (Attr $r.resource 'deployment.environment.name'), (Attr $r.resource 'site.id'), (Attr $r.resource 'node.role')
if (-not $agg[$key]) { $agg[$key] = @{ n = 0; names = @{}; newest = 0 } }
$a = $agg[$key]
if ($sig -eq 'metrics') {
foreach ($sm in $r.scopeMetrics) { foreach ($m in $sm.metrics) { $a.n++; $a.names[$m.name] = 1
foreach ($dp in @($m.sum.dataPoints) + @($m.gauge.dataPoints) + @($m.histogram.dataPoints)) { if ($dp -and [long]$dp.timeUnixNano -gt $a.newest) { $a.newest = [long]$dp.timeUnixNano } } } }
} elseif ($sig -eq 'traces') {
foreach ($ss in $r.scopeSpans) { foreach ($sp in $ss.spans) { $a.n++; $a.names[$sp.name] = 1; if ([long]$sp.endTimeUnixNano -gt $a.newest) { $a.newest = [long]$sp.endTimeUnixNano } } }
} else {
foreach ($sl in $r.scopeLogs) { foreach ($lr in $sl.logRecords) { $a.n++; if ([long]$lr.timeUnixNano -gt $a.newest) { $a.newest = [long]$lr.timeUnixNano } } }
}
}
}
Say "$sig (last $Tail export requests):"
foreach ($k in ($agg.Keys | Sort-Object)) {
$a = $agg[$k]
$newest = if ($a.newest) { [DateTimeOffset]::FromUnixTimeMilliseconds([long]($a.newest / 1000000)).UtcDateTime.ToString('HH:mm:ssZ') } else { '-' }
$fresh = if ($a.newest -ge $cut) { 'FRESH' } else { 'old' }
" $k | items=$($a.n) distinct=$($a.names.Count) newest=$newest $fresh"
if ($sig -eq 'traces') { " spans: " + (($a.names.Keys | Sort-Object | Select-Object -First 8) -join '; ') }
}
}
try {
$p = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8889/metrics').Content -split "`n"
$jobs = $p | Where-Object { $_ -notmatch '^#' -and $_ -match 'job="([^"]+)"' } | ForEach-Object { if ($_ -match 'job="([^"]+)"') { $Matches[1] } } | Group-Object | Sort-Object Name
Say "prometheus :8889 series per job:"; $jobs | ForEach-Object { " $($_.Name) $($_.Count)" }
} catch { Say "prometheus :8889 FAIL $($_.Exception.Message)" }
try {
$q = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 'http://127.0.0.1:8888/metrics').Content -split "`n"
Say "collector self-metrics:"
$q | Where-Object { $_ -match '^otelcol_(receiver_accepted|receiver_refused|exporter_sent|exporter_send_failed)_\w+' } | ForEach-Object { " " + ($_ -replace 'service_instance_id="[^"]*",?','' -replace 'service_name="[^"]*",?','' -replace 'service_version="[^"]*",?','') }
} catch { Say "self-metrics :8888 FAIL $($_.Exception.Message)" }
foreach ($port in $AppPorts) {
$c = 'FAIL'; foreach ($h in 'localhost', $env:COMPUTERNAME) { try { $c = (Invoke-WebRequest -UseBasicParsing -TimeoutSec 10 "http://${h}:$port/metrics").StatusCode; break } catch {} }
" app :$port /metrics $c"
}
@@ -0,0 +1,80 @@
# ZbOtelCollector: per-box OpenTelemetry collector for the wonder tiers (scadaproj#54).
# One instance on each of wonder-app-vd03 (DEV), wonder-app-vq03 and wonder-app-vq04 (QA),
# installed at E:\ApiInstall\OtelCollector as the NSSM service ZbOtelCollector.
#
# Every listener binds 127.0.0.1 only: the apps on the same box export to http://127.0.0.1:4317,
# and nothing is reachable from the network. That is why the receivers have no auth and no TLS.
# Do not change a bind address to 0.0.0.0 without adding auth + TLS first (see README.md).
#
# ZB_TIER (DEV | QA) is set in the service environment and stamped on every signal as
# deployment.environment.name, so files copied off two tiers can still be told apart.
#
# Retention: each file exporter rotates at 100 MB, keeps 5 backups and drops backups older
# than 7 days, so each signal is capped at about 600 MB and the three together at about 1.8 GB.
extensions:
health_check:
endpoint: 127.0.0.1:13133
receivers:
otlp:
protocols:
grpc:
endpoint: 127.0.0.1:4317
http:
endpoint: 127.0.0.1:4318
processors:
memory_limiter:
check_interval: 5s
limit_mib: 256
spike_limit_mib: 64
resource/tier:
attributes:
- key: deployment.environment.name
value: ${env:ZB_TIER}
action: upsert
batch: {}
exporters:
file/metrics:
path: E:/ApiInstall/OtelCollector/data/metrics.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
file/traces:
path: E:/ApiInstall/OtelCollector/data/traces.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
file/logs:
path: E:/ApiInstall/OtelCollector/data/logs.jsonl
rotation: { max_megabytes: 100, max_days: 7, max_backups: 5 }
prometheus:
endpoint: 127.0.0.1:8889
resource_constant_labels:
included: [service.name, service.namespace, service.instance.id, site.id, node.role, host.name, deployment.environment.name]
metric_expiration: 15m
service:
extensions: [health_check]
telemetry:
logs:
level: info
metrics:
level: basic
readers:
- pull:
exporter:
prometheus:
host: 127.0.0.1
port: 8888
pipelines:
metrics:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/metrics, prometheus]
traces:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/traces]
logs:
receivers: [otlp]
processors: [memory_limiter, resource/tier, batch]
exporters: [file/logs]
@@ -0,0 +1,113 @@
{
"ScadaBridge": {
"Node": {
"Role": "Central",
"NodeName": "central-a",
"NodeHostname": "10.220.153.205",
"RemotingPort": 8081
},
"_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.",
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@10.220.153.205:8081",
"akka.tcp://scadabridge@10.220.153.210:8081"
],
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1,
"BootstrapGuard": {
"Enabled": true
}
},
"Database": {
"ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}"
},
"_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.",
"Security": {
"Ldap": {
"Enabled": true,
"Server": "uswap-vdc-nam1.nam.zimmer.com",
"Port": 636,
"Transport": "Ldaps",
"AllowInsecure": false,
"FallbackServers": [
"uswap-vdc-nam2.nam.zimmer.com:636"
],
"SearchBase": "DC=nam,DC=zimmer,DC=com",
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
"UserNameAttribute": "sAMAccountName",
"DisplayNameAttribute": "name",
"GroupAttribute": "memberOf",
"BindMode": "ServiceAccount"
},
"JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}",
"JwtExpiryMinutes": 25,
"IdleTimeoutMinutes": 60,
"RequireHttpsCookie": false,
"ForwardedHeaders": {
"Enabled": false
}
},
"_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).",
"Communication": {
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
"QueryTimeout": "00:00:30",
"TransportHeartbeatInterval": "00:00:05",
"TransportFailureThreshold": "00:00:15",
"CentralFetchBaseUrl": "http://10.220.153.205:8085"
},
"ManagementService": {
"CommandTimeout": "00:00:30",
"LongRunningCommandTimeout": "00:05:00"
},
"HealthMonitoring": {
"ReportInterval": "00:00:30",
"OfflineTimeout": "00:01:00"
},
"InboundApi": {
"DefaultMethodTimeout": "00:00:30",
"ApiKeyStore": {
"Provider": "SqlServer"
},
"StandbyForwardPeerEndpoints": [
"http://10.220.153.210:8085"
]
},
"Notification": {
"SmtpServer": "REPLACE_WITH_SMTP_HOST",
"SmtpPort": 25,
"AuthMode": "None",
"FromAddress": "scada-notifications@zimmer.com"
},
"NotificationOutbox": {
"DispatchInterval": "00:00:10",
"DispatchBatchSize": 100,
"StuckAgeThreshold": "00:10:00",
"TerminalRetention": "365.00:00:00",
"PurgeInterval": "1.00:00:00",
"DeliveredKpiWindow": "00:01:00"
},
"Logging": {
"MinimumLevel": "Information"
}
},
"Secrets": {
"Store": {
"Provider": "SqlServer"
}
},
"Serilog": {
"WriteTo": [
{},
{
"Args": {
"path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log"
}
}
]
}
}
@@ -0,0 +1,113 @@
{
"ScadaBridge": {
"Node": {
"Role": "Central",
"NodeName": "central-b",
"NodeHostname": "10.220.153.210",
"RemotingPort": 8081
},
"_comment_Node": "PROD (plan 2026-11-19): central pair on wonder-app-vp03 (10.220.153.205) / vp04 (10.220.153.210). The site pair is on other boxes, so CentralGrpcPort stays at its default 8083 (QA needed 8086 only because it co-locates a site node). IPv4 literals as on QA: StartupValidator compares SeedNodes[0] with NodeHostname without DNS canonicalisation.",
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@10.220.153.210:8081",
"akka.tcp://scadabridge@10.220.153.205:8081"
],
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1,
"BootstrapGuard": {
"Enabled": true
}
},
"Database": {
"ConfigurationDb": "${SCADABRIDGE_CONFIGURATIONDB_CONNECTION_STRING}"
},
"_comment_Database": "Overridden by the service env ScadaBridge__Database__ConfigurationDb (QA pattern). MachineDataDb is gone since 3808b49f.",
"Security": {
"Ldap": {
"Enabled": true,
"Server": "uswap-vdc-nam1.nam.zimmer.com",
"Port": 636,
"Transport": "Ldaps",
"AllowInsecure": false,
"FallbackServers": [
"uswap-vdc-nam2.nam.zimmer.com:636"
],
"SearchBase": "DC=nam,DC=zimmer,DC=com",
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
"UserNameAttribute": "sAMAccountName",
"DisplayNameAttribute": "name",
"GroupAttribute": "memberOf",
"BindMode": "ServiceAccount"
},
"JwtSigningKey": "${SCADABRIDGE_JWT_SIGNING_KEY}",
"JwtExpiryMinutes": 25,
"IdleTimeoutMinutes": 60,
"RequireHttpsCookie": false,
"ForwardedHeaders": {
"Enabled": false
}
},
"_comment_Security": "Login required from the first start, NAM AD over LDAPS (plan \u00a73A, D-P5 svc_igntadmin, D-P7 plain HTTP). JwtSigningKey is overridden by the env ScadaBridge__Security__JwtSigningKey, the SAME value on both centrals. No ScadaBridge__Security__Ldap__* env entry: it would override the ${secret:} reference. Roles come from dbo.LdapGroupMappings (_PROD rows, sql/sb-role-mappings-prod.sql).",
"Communication": {
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
"QueryTimeout": "00:00:30",
"TransportHeartbeatInterval": "00:00:05",
"TransportFailureThreshold": "00:00:15",
"CentralFetchBaseUrl": "http://10.220.153.210:8085"
},
"ManagementService": {
"CommandTimeout": "00:00:30",
"LongRunningCommandTimeout": "00:05:00"
},
"HealthMonitoring": {
"ReportInterval": "00:00:30",
"OfflineTimeout": "00:01:00"
},
"InboundApi": {
"DefaultMethodTimeout": "00:00:30",
"ApiKeyStore": {
"Provider": "SqlServer"
},
"StandbyForwardPeerEndpoints": [
"http://10.220.153.205:8085"
]
},
"Notification": {
"SmtpServer": "REPLACE_WITH_SMTP_HOST",
"SmtpPort": 25,
"AuthMode": "None",
"FromAddress": "scada-notifications@zimmer.com"
},
"NotificationOutbox": {
"DispatchInterval": "00:00:10",
"DispatchBatchSize": 100,
"StuckAgeThreshold": "00:10:00",
"TerminalRetention": "365.00:00:00",
"PurgeInterval": "1.00:00:00",
"DeliveredKpiWindow": "00:01:00"
},
"Logging": {
"MinimumLevel": "Information"
}
},
"Secrets": {
"Store": {
"Provider": "SqlServer"
}
},
"Serilog": {
"WriteTo": [
{},
{
"Args": {
"path": "E:\\ApiInstall\\ScadaBridge\\central\\logs\\scadabridge-central-.log"
}
}
]
}
}
@@ -0,0 +1,89 @@
{
"ScadaBridge": {
"Node": {
"Role": "Site",
"NodeName": "node-a",
"NodeHostname": "10.201.75.86",
"SiteId": "__Q8_SITE_ID__",
"RemotingPort": 8082,
"GrpcPort": 8083,
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@10.201.75.86:8082",
"akka.tcp://scadabridge@10.201.75.87:8082"
],
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1,
"BootstrapGuard": {
"Enabled": true
}
},
"_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.",
"Database": {
"SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db"
},
"DataConnection": {
"ReconnectInterval": "00:00:05",
"TagResolutionRetryInterval": "00:00:10",
"WriteTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db"
},
"Communication": {
"CentralGrpcEndpoints": [
"http://10.220.153.205:8083",
"http://10.220.153.210:8083"
],
"_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-<SiteId>.",
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
"QueryTimeout": "00:00:30",
"TransportHeartbeatInterval": "00:00:05",
"TransportFailureThreshold": "00:00:15"
},
"HealthMonitoring": {
"ReportInterval": "00:00:30",
"OfflineTimeout": "00:01:00"
},
"SiteEventLog": {
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db",
"RetentionDays": 30,
"MaxStorageMb": 1024,
"PurgeScheduleCron": "0 2 * * *"
},
"Notification": {},
"Logging": {
"MinimumLevel": "Information"
}
},
"Serilog": {
"WriteTo": [
{},
{
"Args": {
"path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log"
}
}
]
},
"LocalDb": {
"Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db",
"Replication": {
"PeerAddress": "http://10.201.75.87:8083",
"MaxBatchSize": 16,
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
},
"AuditLog": {
"SiteWriter": {
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db"
}
}
}
@@ -0,0 +1,88 @@
{
"ScadaBridge": {
"Node": {
"Role": "Site",
"NodeName": "node-b",
"NodeHostname": "10.201.75.87",
"SiteId": "__Q8_SITE_ID__",
"RemotingPort": 8082,
"GrpcPort": 8083,
"MetricsPort": 8084
},
"Cluster": {
"SeedNodes": [
"akka.tcp://scadabridge@10.201.75.87:8082",
"akka.tcp://scadabridge@10.201.75.86:8082"
],
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
"MinNrOfMembers": 1,
"BootstrapGuard": {
"Enabled": true
}
},
"_comment_Cluster": "PROD site pair on wonder-par-vp03 (10.201.75.86) / par-vp04 (10.201.75.87), a separate Akka cluster from the centrals. SeedNodes[0] MUST be this node.",
"Database": {
"SiteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site.db"
},
"DataConnection": {
"ReconnectInterval": "00:00:05",
"TagResolutionRetryInterval": "00:00:10",
"WriteTimeout": "00:00:30"
},
"StoreAndForward": {
"SqliteDbPath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\store-and-forward.db"
},
"Communication": {
"CentralGrpcEndpoints": [
"http://10.220.153.205:8083",
"http://10.220.153.210:8083"
],
"_comment": "Both centrals on their default CentralGrpcPort 8083, same list and order on both site nodes. This crosses the 10.201.75.x -> 10.220.153.x boundary (plan \u00a74 firewall rows). GrpcPsk comes from the service env ScadaBridge__Communication__GrpcPsk; central holds the same value as secret SB-GRPC-PSK-<SiteId>.",
"DeploymentTimeout": "00:02:00",
"LifecycleTimeout": "00:00:30",
"QueryTimeout": "00:00:30",
"TransportHeartbeatInterval": "00:00:05",
"TransportFailureThreshold": "00:00:15"
},
"HealthMonitoring": {
"ReportInterval": "00:00:30",
"OfflineTimeout": "00:01:00"
},
"SiteEventLog": {
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site_events.db",
"RetentionDays": 30,
"MaxStorageMb": 1024,
"PurgeScheduleCron": "0 2 * * *"
},
"Notification": {},
"Logging": {
"MinimumLevel": "Information"
}
},
"Serilog": {
"WriteTo": [
{},
{
"Args": {
"path": "E:\\ApiInstall\\ScadaBridge\\site\\logs\\scadabridge-site-.log"
}
}
]
},
"LocalDb": {
"Path": "E:\\ApiInstall\\ScadaBridge\\site\\data\\site-localdb.db",
"Replication": {
"MaxBatchSize": 16,
"MaxOplogRows": 250000,
"MaxOplogAge": "2.00:00:00"
}
},
"AuditLog": {
"SiteWriter": {
"DatabasePath": "E:\\ApiInstall\\ScadaBridge\\site\\data\\auditlog.db"
}
}
}
@@ -0,0 +1,110 @@
# PROD ScadaBridge first install of ONE node (plan Phase 5 = Central on vp03/vp04, Phase 6 = Site on par-vp03/04).
# Run ON the box, only after the owner's go for that phase. Never starts the service.
# -Mode dry : verify the zip, pick this box's config, check placeholders + values, expand into .\dry\, print the
# service definition (env names only). Creates nothing outside this stage folder.
# -Mode real : expand into E:\ApiInstall\ScadaBridge\{app,cli}, put this node's appsettings.<Role>.json in place,
# create <role>\{data,logs}, the KEK (or -KekFrom; ONE scadabridge.kek for all four nodes), and the
# ScadaBridge-<Role> service (LocalSystem, auto, env REG_MULTI_SZ, recovery 5s/5s/10s, failureflag 1).
# -ValuesDir holds one <KEY>.txt per '@file' line of sb-<role>.env; never printed; the caller deletes it afterwards.
param(
[Parameter(Mandatory)][ValidateSet('Central','Site')][string]$Role,
[Parameter(Mandatory)][ValidateSet('dry','real')][string]$Mode,
[string]$ValuesDir,
[string]$KekFrom
)
$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'
$stage = $PSScriptRoot
$zip = Join-Path $stage 'scadabridge-2e0ce81e.zip'
$zipSha = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23'
$root = 'E:\ApiInstall\ScadaBridge'
$kekDir = 'C:\ProgramData\ZB\kek'
$kek = "$kekDir\scadabridge.kek"
$svc = "ScadaBridge-$Role"
function Say($m) { "[{0:HH:mm:ss}] {1}" -f (Get-Date), $m }
function Fail($m) { throw "ABORT: $m" }
$boxMap = @{ 'WONDER-APP-VP03' = @('vp03','Central'); 'WONDER-APP-VP04' = @('vp04','Central')
'WONDER-PAR-VP03' = @('parvp03','Site'); 'WONDER-PAR-VP04' = @('parvp04','Site') }
$b = $boxMap[$env:COMPUTERNAME.ToUpperInvariant()]
if (-not $b) { Fail "unknown box $env:COMPUTERNAME" }
if ($b[1] -ne $Role) { Fail "$env:COMPUTERNAME is a $($b[1]) box in the PROD topology, not $Role" }
Say "install-sb role=$Role mode=$Mode box=$env:COMPUTERNAME"
# 1. artifact
$h = (Get-FileHash $zip -Algorithm SHA256).Hash
if ($h -ne $zipSha) { Fail "zip sha256 $h != $zipSha" }
Say 'zip sha256 ok'
# 2. this node's config
$cfg = Join-Path $stage "appsettings.$Role.json.prod-$($b[0])"
$cfgText = [IO.File]::ReadAllText($cfg)
if ($cfgText -match '__Q\d+_[A-Z_]+__') { Fail "unresolved placeholder $($Matches[0]) in $(Split-Path $cfg -Leaf)" }
if ($cfgText -match '_(DEV|QA)"') { Fail 'a _DEV/_QA name in a PROD config' }
$null = $cfgText | ConvertFrom-Json
Say "config $(Split-Path $cfg -Leaf) ok"
# 3. env
$lines = Get-Content (Join-Path $stage "sb-$($Role.ToLowerInvariant()).env")
$envList = New-Object System.Collections.Generic.List[string]
foreach ($l in $lines) {
if ($l -match '^\s*(#|$)') { continue }
$k, $v = $l -split '=', 2
if ($v -match '__Q\d+_') { Fail "unresolved placeholder in env $k" }
if ($v -eq '@file') {
if (-not $ValuesDir) { Fail "$k needs -ValuesDir" }
$f = Join-Path $ValuesDir "$k.txt"
if (-not (Test-Path -LiteralPath $f)) { Fail "missing value file for $k" }
$v = [IO.File]::ReadAllText($f).TrimEnd("`r", "`n")
if (-not $v) { Fail "empty value file for $k" }
if ($k -like '*JwtSigningKey' -and $v.Length -lt 32) { Fail 'JwtSigningKey shorter than 32 characters' }
}
$envList.Add("$k=$v")
}
Say "env entries: $($envList.Count) ($((($envList | ForEach-Object { ($_ -split '=')[0] }) -join ', ')))"
# 4. layout
$base = if ($Mode -eq 'dry') { Join-Path $stage 'dry' } else { $root }
$app = Join-Path $base 'app'; $cli = Join-Path $base 'cli'
if ($Mode -eq 'real') {
if (Test-Path $app) { Fail "$app already exists - first-install script only" }
if (Get-Service $svc -ErrorAction SilentlyContinue) { Fail "service $svc already exists" }
} elseif (Test-Path $base) { Remove-Item $base -Recurse -Force }
New-Item -ItemType Directory -Force $base | Out-Null
Expand-Archive -LiteralPath $zip -DestinationPath $base
Copy-Item $cfg (Join-Path $app "appsettings.$Role.json") -Force
$roleDir = Join-Path $base $Role.ToLowerInvariant()
New-Item -ItemType Directory -Force (Join-Path $roleDir 'data'), (Join-Path $roleDir 'logs') | Out-Null
$exe = Join-Path $app 'ZB.MOM.WW.ScadaBridge.Host.exe'
Say "expanded to ${base}: app $(@(Get-ChildItem $app -Recurse -File).Count) files, cli $(@(Get-ChildItem $cli -Recurse -File).Count) files; Host $((Get-Item $exe).VersionInfo.ProductVersion)"
# 5. KEK
if ($Mode -eq 'real') {
New-Item -ItemType Directory -Force $kekDir | Out-Null
& icacls.exe $kekDir /inheritance:r /grant:r 'BUILTIN\Administrators:(OI)(CI)F' 'NT AUTHORITY\SYSTEM:(OI)(CI)F' | Out-Null
if (Test-Path $kek) { Say "KEK exists, kept" }
elseif ($KekFrom) { Copy-Item $KekFrom $kek; Say 'KEK copied from -KekFrom' }
else {
if ($b[0] -ne 'vp03') { Fail 'only vp03 generates scadabridge.kek; every other node takes the same file via -KekFrom' }
$r = New-Object byte[] 32; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($r); [IO.File]::WriteAllBytes($kek, $r); Say 'KEK generated'
}
if ((Get-Item $kek).Length -ne 32) { Fail 'KEK file is not 32 bytes' }
Say "KEK sha256 $((Get-FileHash $kek).Hash.Substring(0,8)) (must match on all four nodes)"
}
# 6. service, never started
$disp = "ScadaBridge $Role Node"
if ($Mode -eq 'dry') {
Say "[DRYRUN] sc.exe create $svc binPath= `"$exe`" start= auto obj= LocalSystem DisplayName= `"$disp`""
Say "[DRYRUN] HKLM\...\Services\$svc Environment = REG_MULTI_SZ, $($envList.Count) entries (values hidden)"
Say "[DRYRUN] sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000; sc.exe failureflag $svc 1"
} else {
& sc.exe create $svc binPath= "`"$(Join-Path $root 'app\ZB.MOM.WW.ScadaBridge.Host.exe')`"" start= auto obj= LocalSystem DisplayName= $disp | Out-Null
if ($LASTEXITCODE -ne 0) { Fail "sc create exited $LASTEXITCODE" }
Set-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc" -Name Environment -Type MultiString -Value $envList.ToArray()
& sc.exe failure $svc reset= 86400 actions= restart/5000/restart/5000/restart/10000 | Out-Null
& sc.exe failureflag $svc 1 | Out-Null
$n = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$svc").Environment).Count
if ($n -ne $envList.Count) { Fail "env read-back $n entries, expected $($envList.Count)" }
Say "service $svc created STOPPED, env $n entries"
if ($Role -eq 'Central') { Say 'Next: ConfigDb (sql\), role rows, zbsecrets schema, seed nam-ldap-svc-password ONCE, then start vp03, then vp04.' }
else { Say 'Next: SB-GRPC-PSK-<siteId> in central store + Site entity; start BOTH site nodes together (plan Phase 6).' }
}
Say "DONE mode=$Mode"
@@ -0,0 +1,15 @@
# ScadaBridge-Central service env (REG_MULTI_SZ Environment) for PROD vp03 + vp04. QA key set minus MachineDataDb and
# the LDAP password (§3A.3) and minus SitePsks (central reads SB-GRPC-PSK-<siteId> from its store, Phase 6).
# @file = <ValuesDir>\<KEY>.txt at install time. JwtSigningKey and ApiKeyPepper MUST be the same on both centrals.
SCADABRIDGE_CONFIG=Central
ASPNETCORE_URLS=http://+:8085
ScadaBridge__Database__ConfigurationDb=@file
ScadaBridge__Security__JwtSigningKey=@file
ScadaBridge__InboundApi__ApiKeyPepper=@file
ScadaBridge__InboundApi__ApiKeyStore__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\inbound-api-keys.sqlite
Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\central\data\secrets.db
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek
Secrets__SqlServer__ConnectionString=@file
ScadaBridge__Telemetry__Exporter=Otlp
ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
@@ -0,0 +1,11 @@
# ScadaBridge-Site service env for PROD par-vp03 + par-vp04 (QA key set). Both values MUST be identical on both site
# nodes; GrpcPsk also goes into central's store as SB-GRPC-PSK-<siteId>. Never reuse the PSK as the LocalDb key.
# The KEK file must be the SAME scadabridge.kek as the centrals (deployed secrets are central-encrypted).
SCADABRIDGE_CONFIG=Site
ScadaBridge__Communication__GrpcPsk=@file
LocalDb__Replication__ApiKey=@file
Secrets__SqlitePath=E:\ApiInstall\ScadaBridge\site\data\secrets.db
Secrets__MasterKey__Source=File
Secrets__MasterKey__FilePath=C:\ProgramData\ZB\kek\scadabridge.kek
ScadaBridge__Telemetry__Exporter=Otlp
ScadaBridge__Telemetry__OtlpEndpoint=http://127.0.0.1:4317
@@ -0,0 +1,20 @@
-- PROD ScadaBridge LDAP group -> role rows (plan §3A.3, D-P6). Run in the PROD ScadaBridge DB after the schema script
-- (sqlcmd -I -b). Idempotent; refuses if a group is already mapped to a different role. Only _PROD names (§3A.2).
-- Later changes go through the CLI (audited). Ids 1-5 are the migration's SCADA-* seed rows; they stay untouched.
SET QUOTED_IDENTIFIER ON;
SET XACT_ABORT ON;
BEGIN TRAN;
DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL);
INSERT @t VALUES
(N'UG_WW_APP_ADMIN_PROD', N'Administrator'),
(N'UG_WW_APP_IDE_PROD', N'Designer'),
(N'UG_WW_APP_IDE_RO_PROD', N'Viewer'),
(N'UG_WW_BTMGR_RO_PROD', N'Viewer'),
(N'UG_WW_HIST_ADMIN_PROD', N'Deployer'); -- interim until UG_WW_APP_DEPLOY_PROD exists (D-P6)
IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role)
THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1;
INSERT dbo.LdapGroupMappings (LdapGroupName, Role)
SELECT t.LdapGroupName, t.Role FROM @t t
WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName);
COMMIT;
SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id;
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,44 @@
-- ScadaBridge Central shared secret store (Secrets:Store:Provider=SqlServer), schema "zbsecrets", version 1.
-- Verbatim DDL from ZB.MOM.WW.Secrets.Replicator.SqlServer SqlServerSecretsSchema.CreateSchemaDdl (unchanged since 0.2.0,
-- so identical in the 0.6.2 package ScadaBridge 2e0ce81e pins), plus the version-row MERGE its migrator writes.
-- WHY: the app creates this at first start, but nam-ldap-svc-password must be seeded BEFORE the first start (§3A.4) and the
-- headless CLI runs no migrator. Idempotent; run in the ScadaBridge DB after scadabridge-schema-2e0ce81e.sql.
SET QUOTED_IDENTIFIER ON;
IF NOT EXISTS (SELECT 1 FROM sys.schemas WHERE name = N'zbsecrets')
EXEC(N'CREATE SCHEMA [zbsecrets]');
GO
IF OBJECT_ID(N'[zbsecrets].[schema_version]', N'U') IS NULL
CREATE TABLE [zbsecrets].[schema_version] (
id INT NOT NULL PRIMARY KEY CHECK (id = 1),
version INT NOT NULL,
applied_utc NVARCHAR(33) NOT NULL
);
IF OBJECT_ID(N'[zbsecrets].[secret]', N'U') IS NULL
CREATE TABLE [zbsecrets].[secret] (
name NVARCHAR(450) NOT NULL PRIMARY KEY,
description NVARCHAR(MAX) NULL,
content_type NVARCHAR(64) NOT NULL,
ciphertext VARBINARY(MAX) NOT NULL,
nonce VARBINARY(MAX) NOT NULL,
tag VARBINARY(MAX) NOT NULL,
wrapped_dek VARBINARY(MAX) NOT NULL,
wrap_nonce VARBINARY(MAX) NOT NULL,
wrap_tag VARBINARY(MAX) NOT NULL,
kek_id NVARCHAR(256) NOT NULL,
revision BIGINT NOT NULL CONSTRAINT DF_secret_revision DEFAULT 0,
is_deleted BIT NOT NULL CONSTRAINT DF_secret_is_deleted DEFAULT 0,
deleted_utc NVARCHAR(33) NULL,
created_utc NVARCHAR(33) NOT NULL,
updated_utc NVARCHAR(33) NOT NULL,
created_by NVARCHAR(256) NULL,
updated_by NVARCHAR(256) NULL
);
IF NOT EXISTS (SELECT 1 FROM sys.indexes WHERE name = N'ix_secret_updated_utc' AND object_id = OBJECT_ID(N'[zbsecrets].[secret]'))
CREATE INDEX ix_secret_updated_utc ON [zbsecrets].[secret] (updated_utc);
MERGE [zbsecrets].[schema_version] AS target
USING (SELECT 1 AS id, 1 AS version, CONVERT(nvarchar(33), FORMAT(SYSUTCDATETIME(), 'yyyy-MM-ddTHH:mm:ss.fffffff') + N'+00:00') AS applied_utc) AS source
ON target.id = source.id
WHEN NOT MATCHED THEN INSERT (id, version, applied_utc) VALUES (1, source.version, source.applied_utc);
GO
SELECT s.name AS [schema], t.name AS [table] FROM sys.tables t JOIN sys.schemas s ON s.schema_id = t.schema_id WHERE s.name = N'zbsecrets';
SELECT id, version, applied_utc FROM [zbsecrets].[schema_version];