deploy(ldap): QA vq03/vq04 NAM AD cutover executed with the DEV builds — HG, mxgw, ScadaBridge, OtOpcUa (record + scripts)

Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
Joseph Doherty
2026-11-19 16:14:01 -05:00
parent 386c999720
commit 9789b01acd
14 changed files with 652 additions and 2 deletions
@@ -0,0 +1,98 @@
# 2026-11-19: QA vq03/vq04 NAM AD LDAP cutover + DEV builds (HistorianGateway, MxAccessGateway, ScadaBridge, OtOpcUa)
**Owner go (2026-11-19):** "do the qa ldap plan, winrm usage approved". Plan:
[`docs/plans/2026-11-15-nam-ldap-cutover.md`](../../plans/2026-11-15-nam-ldap-cutover.md) §0Q (svc_igntadmin interim, DEV builds,
one window). DEV record: [`../2026-11-19-vd03-nam-ldap/`](../2026-11-19-vd03-nam-ldap/README.md). Umbrella: scadaproj#6.
**Result: all four apps on both QA boxes now run the DEV builds and sign users in against NAM AD** over LDAPS (nam1, nam2
as fallback), bound as `svc_igntadmin@nam.zimmer.com` with the password from `${secret:nam-ldap-svc-password}` in each
app's own store. The GLAuth `multi-role` identity no longer signs in anywhere on QA. **Zero service-account bind
failures** and zero LDAP warnings in either box's OTel log export for the whole window.
| App | Build | Where the LDAP config now lives |
|---|---|---|
| HistorianGateway (per box) | `6365f34` → **`f84af29`** | NSSM env `Ldap__*` + `Dashboard__GroupToRole__*` |
| MxAccessGateway (per box) | `76a71bc8` → **`e74918cc`** | NSSM env `MxGateway__Ldap__*` + `MxGateway__Dashboard__GroupToRole__*`; base `app\appsettings.json` lost `GwAdmin`/`GwReader` |
| ScadaBridge (pair) | `061432c8` → **`2e0ce81e`** | each Central's `app\appsettings.Central.json` (`ScadaBridge:Security:Ldap`); roles in `LdapGroupMappings` |
| OtOpcUa (Warm pair) | `76e73e7d` → **`cdf433a3`** | service env `Security__Ldap__*` (+ `Security__Cookie__RequireHttpsCookie=false`) |
Every artifact is the one DEV runs, pulled from vd03's stage dirs over SMB and re-hashed (HG zip `1595a828…`, mxgw
`f0999b75…`, OtOpcUa `c89ba972…`, ScadaBridge `6c706b0a…`). No app had a schema change.
## Steps (EST)
| Time | Step |
|---|---|
| 15:29 | **Phase 0** (`preflight.ps1`, both boxes): nam1/nam2 reachable on 636 + 389; `zimmer-rootca` in `LocalMachine\Root` (`78D941D7…`, exp 2036); LDAPS chains validate (TLS 1.3, no policy errors). Every service key and `appsettings*.json` → `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04), protected DACL. (The first run died on a parsing bug after one `reg export`; its `…T152920`/`…T152930` dirs hold only that file.) OtOpcUa Primary **vq04**, ScadaBridge active central **vq03**. HG and mxgw already had `RequireHttpsCookie=false` in env; OtOpcUa had no cookie setting |
| 15:35 | Staged on both boxes: the four builds under `E:\psrem-stage\{hg-f84af29,mxgw-e74918cc,otopcua-cdf433a3,sb-2e0ce81e}`, the Secrets CLI 0.6.6 at `E:\ApiInstall\_tools\secret-0.6.6\`. Hashes checked on box |
| 15:37 | **ScadaBridge role rows by CLI** as GLAuth `multi-role` against the active central (audited path): ids 6–10, `UG_WW_APP_ADMIN_QA`→Administrator, `UG_WW_APP_IDE_QA`→Designer, `UG_WW_APP_IDE_RO_QA`→Viewer, **`UG_WW_HIST_ADMIN_QA`→Deployer (interim)**, `UG_WW_BTMGR_RO_QA`→Viewer |
| 15:41 | **Secret seeded** (`seed-secret.ps1`; each store's path and KEK read from its own service env): HG, mxgw, OtOpcUa on each box (SQLite, KEK files under `C:\ProgramData\ZB\kek\`), ScadaBridge **once**, into the shared SQL Server store (`zbsecrets` schema; now 1 row). Read-back matched in all 7; the value file was ACL'd and deleted. Both Centrals' `scadabridge.kek` are identical (`A3BD293A…`). See finding 2 for the SQL Server seeding |
| 15:43–15:48 | **HistorianGateway** vq03 then vq04: `env-hg.ps1` (34 → 41 entries: NAM `Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then the f84af29 pack (`preflight.ps1` 0 FAIL / 0 WARN incl. a side-by-side boot with the new env, `swap.ps1`, `verify.ps1` 0 FAIL). Downtime **6.6 s / 7.7 s** |
| 15:56–15:59 | **MxAccessGateway** vq03 then vq04: `env-mxgw.ps1` (33 → 38: NAM `MxGateway__Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then `mxgw-upgrade.tmpl.ps1` (puts the NAM base `appsettings.json` in place, then the repo upgrade script with the `START_PENDING` try/catch patch from vd03; it carries the live appsettings into the new `app\`). Dry run first on vq03. Env fingerprint unchanged through stop/swap/start/60 s stability on both |
| 16:00 | **ScadaBridge** `prep` both nodes; DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak` (VERIFYONLY ok; 65 migrations) |
| 16:00:53–16:04:14 | ScadaBridge window (`sb-qa.tmpl.ps1`): stop Site vq03, vq04, Central vq03, vq04; swap both (binaries + each node's own NAM `appsettings.Central.json`; `ScadaBridge__Security__Ldap__ServiceAccountPassword` removed, 15 → 14 env entries); start Central vq03 (16:03:17), vq04 (16:03:37), Site vq03 (16:03:55), vq04 (16:04:14). **Site data path ~2 min 17 s, central API ~55 s** |
| 16:05–16:10 | **OtOpcUa** (`otopcua-prepare/swap2/start.ps1`, the 11-13 QA pack retargeted): prepare both (1410 files, deps ok, ConfigDb `OtOpcUaConfig_pre-cdf433a3_20261119.bak` + VERIFYONLY on vq03, 9 migrations, isolation `-Verify` OK); swap the Standby vq03 (`-NoStart`), `env-otopcua.ps1` (51 → 61); swap vq04 (stop 16:08:33), env (50 → 60); start vq04 (ready 16:09:29, 37.8 s), vq03 (ready 16:09:49, 8 s). Full isolation `-Verify` **OK 21/21** on both. **Pair outage ~56 s** |
## Mappings now on QA
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
ScadaBridge still holds the five `SCADA-*` seed rows (ids 1–5); they grant nothing in NAM.
## Verification
| Check | HG `:5220` | mxgw `:5130` | ScadaBridge `:8085` | OtOpcUa `:9000` |
|---|---|---|---|---|
| Anonymous `GET /` → login | 302 both | 302 both | 302 | 302 both |
| Bogus user refused | ✓ both | ✓ both | ✓ (GLAuth `multi-role` refused too, UI + CLI) | ✓ both |
| `dohertj2` signs in | ✓ both | ✓ both | ✓ cookie; `/auth/token` 200 roles **Administrator+Designer+Deployer**; CLI (HTTP Basic) `instance list` (11) + admin `role-mapping list` | ✓ cookie, `GET /` 200 (no redirect loop); `/auth/token` 200 roles **Viewer+Designer+Administrator** |
| After | pack `verify.ps1` 0 FAIL both; `ReadAggregate` gate as expected | `/health/ready` 200, env unchanged | central + site ready on both; active central vq03; site-a both nodes online; **MxGateway Connected 2196/2196 Good**; **WarmCanary Connected (to the Primary, vq04) 2/2 Good** after the OtOpcUa restart; LocalDb replication connected | **vq04 Primary 250 (`warm_role` 1), vq03 Standby 90**; `localdb_oplog_depth` 0 both; script workers up (shards 0, 1) both |
Overview `:5320` untouched. **Not done, by owner instruction:** DC-failover checks and firewall rules. **Not done:** the
optional oswaltk Viewer-only login (step 8; needs that user).
## Findings
1. **HG pack assumed QA was on `fa9cf31`.** QA has run `6365f34` since 11-13. The pack's QA source SHA was patched
(preflight/swap) before the run; nothing else in the pack needed changing.
2. **The headless Secrets CLI cannot write the SQL Server store.** `secret set` registers only `AddZbSecrets` (SQLite);
the interactive console does honour `Secrets:SqlServer`, but needs a TTY. ScadaBridge's Central uses
`Secrets:Store:Provider=SqlServer`, so a one-off CLI build (0.6.6 + "if `Secrets:Store:Provider=SqlServer`, call
`AddZbSecretsSqlServerStore`", mirroring ScadaBridge's own registration) seeded it, staged at
`E:\ApiInstall\_tools\secret-0.6.6-sql\` on vq03. **Follow-up:** make the headless CLI honour the provider key the
same way, in `ZB.MOM.WW.Secrets`.
3. **mxgw on QA keeps its LDAP settings in the NSSM env**, not in `appsettings.json` (unlike vd03). The base file's
`GwAdmin`/`GwReader` merge with the env `GroupToRole`, so they were still removed from the file. The repo upgrade
script carries the live `appsettings*.json` forward, so future redeploys keep the edit.
4. **OtOpcUa's first start after a pair stop took 37.8 s** (vq04, alone in the mesh); the pair outage was ~56 s against
~37 s on 11-13. The swap itself had no `file in use` retry this time (the #908 fix is in the cdf433a3 installer).
5. Noise seen, not caused here: OtOpcUa "Mesh PSK secret `ot-mesh-psk-dev` for cluster DEV is absent" (QA has no DEV
cluster mesh peer); HG plaintext/relative-path production-readiness warnings.
## Rollback (per app)
Keys and files: `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04) (`reg-<service>.reg`, `<app>_app_appsettings*.json`).
- **HG:** the pack's `rollback.ps1` (→ `app.bak.20261119T154443` vq03 / `…T154740` vq04), then `reg import reg-HistorianGateway.reg`, restart.
- **mxgw:** `Upgrade-MxGatewayService.ps1 -Rollback` (stamps `20261119-155641` vq03 / `20261119-155819` vq04), then `reg import reg-MxAccessGw.reg` and restore `MxAccessGateway_app_appsettings.json`, restart.
- **ScadaBridge:** `rollback-swap` action (`app.bak.20261119T1600`, Central env from `_envbak-ScadaBridge-Central-20261119T1600.reg`), start Central vq03, vq04, then the Sites. No DB restore needed. Keep the `UG_WW_*` rows.
- **OtOpcUa:** stop both; swap `app.bak.20261119-pre-cdf433a3` back; `reg import reg-OtOpcUaHost.reg` on each; start both.
The seeded secrets are inert once nothing references them. The box-local GLAuth (`ScadaBridge-LDAP`) is still running
until the soak ends.
## Automation identity
QA automation now uses **`NAM\dohertj2`** (HTTP Basic / `--password-file`; Keychain `winrm-cli/vq03`). No checked-in QA
script used `multi-role`; the remaining `multi-role` users are the local Docker rig scripts, which stay on GLAuth.
## Files
`preflight.ps1`, `seed-secret.ps1`, `env-edit.ps1` (+ rendered `env-hg.ps1`, `env-mxgw.ps1`, `env-otopcua.ps1`),
`mxgw-upgrade.tmpl.ps1`, `sb-qa.tmpl.ps1`, `otopcua-prepare.ps1`, `otopcua-swap.ps1`, `otopcua-swap2.ps1`,
`otopcua-start.ps1`. Login probes reused `../2026-11-19-vd03-nam-ldap/login-test.sh`, `sb-cli.sh`, `otelgrep.py`.
@@ -0,0 +1,19 @@
# Rewrite one service's env block (NSSM AppEnvironmentExtra or SCM Environment): drop entries whose NAME matches
# $Drop, append $Add. Every other entry stays byte-identical (checked). No restart. Rendered per call.
$ErrorActionPreference = 'Stop'
$Svc = '__SVC__'; $Kind = '__KIND__' # nssm | svc
$Drop = '__DROP__'
$Add = @(__ADD__)
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$Svc"
if ($Kind -eq 'nssm') { $k = "$k\Parameters"; $prop = 'AppEnvironmentExtra' } else { $prop = 'Environment' }
$before = @((Get-ItemProperty $k -Name $prop).$prop)
$kept = @($before | Where-Object { $n = $_.Substring(0, [Math]::Max(0,$_.IndexOf('='))); $n -notmatch $Drop })
$dropped = @($before | Where-Object { $kept -notcontains $_ } | ForEach-Object { $_.Substring(0,$_.IndexOf('=')) })
foreach ($a in $Add) { $n = $a.Substring(0,$a.IndexOf('=')); if (@($kept | Where-Object { $_ -like "$n=*" }).Count) { throw "add collides with kept entry $n" } }
$new = $kept + $Add
Set-ItemProperty -Path $k -Name $prop -Value ([string[]]$new) -Type MultiString
$after = @((Get-ItemProperty $k -Name $prop).$prop)
if ($after.Count -ne $new.Count) { throw 'count mismatch after write' }
for ($i = 0; $i -lt $new.Count; $i++) { if ($after[$i] -cne $new[$i]) { throw "entry $i differs after write" } }
"$Svc ($prop) $($before.Count) -> $($after.Count) entries; dropped: $($dropped -join ', ')"
foreach ($a in $Add) { " + $a" }
@@ -0,0 +1,35 @@
# Rewrite one service's env block (NSSM AppEnvironmentExtra or SCM Environment): drop entries whose NAME matches
# $Drop, append $Add. Every other entry stays byte-identical (checked). No restart. Rendered per call.
$ErrorActionPreference = 'Stop'
$Svc = 'HistorianGateway'; $Kind = 'nssm' # nssm | svc
$Drop = '^(Ldap__|Dashboard__GroupToRole__)'
$Add = @('Ldap__Enabled=true',
'Ldap__Server=uswap-vdc-nam1.nam.zimmer.com',
'Ldap__Port=636',
'Ldap__Transport=Ldaps',
'Ldap__AllowInsecure=false',
'Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636',
'Ldap__SearchBase=DC=nam,DC=zimmer,DC=com',
'Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com',
'Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}',
'Ldap__UserNameAttribute=sAMAccountName',
'Ldap__DisplayNameAttribute=name',
'Ldap__GroupAttribute=memberOf',
'Ldap__BindMode=ServiceAccount',
'Dashboard__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator',
'Dashboard__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer',
'Dashboard__GroupToRole__UG_WW_HIST_ADMIN_QA=Administrator',
'Dashboard__GroupToRole__UG_WW_BTMGR_RO_QA=Viewer')
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$Svc"
if ($Kind -eq 'nssm') { $k = "$k\Parameters"; $prop = 'AppEnvironmentExtra' } else { $prop = 'Environment' }
$before = @((Get-ItemProperty $k -Name $prop).$prop)
$kept = @($before | Where-Object { $n = $_.Substring(0, [Math]::Max(0,$_.IndexOf('='))); $n -notmatch $Drop })
$dropped = @($before | Where-Object { $kept -notcontains $_ } | ForEach-Object { $_.Substring(0,$_.IndexOf('=')) })
foreach ($a in $Add) { $n = $a.Substring(0,$a.IndexOf('=')); if (@($kept | Where-Object { $_ -like "$n=*" }).Count) { throw "add collides with kept entry $n" } }
$new = $kept + $Add
Set-ItemProperty -Path $k -Name $prop -Value ([string[]]$new) -Type MultiString
$after = @((Get-ItemProperty $k -Name $prop).$prop)
if ($after.Count -ne $new.Count) { throw 'count mismatch after write' }
for ($i = 0; $i -lt $new.Count; $i++) { if ($after[$i] -cne $new[$i]) { throw "entry $i differs after write" } }
"$Svc ($prop) $($before.Count) -> $($after.Count) entries; dropped: $($dropped -join ', ')"
foreach ($a in $Add) { " + $a" }
@@ -0,0 +1,34 @@
# Rewrite one service's env block (NSSM AppEnvironmentExtra or SCM Environment): drop entries whose NAME matches
# $Drop, append $Add. Every other entry stays byte-identical (checked). No restart. Rendered per call.
$ErrorActionPreference = 'Stop'
$Svc = 'MxAccessGw'; $Kind = 'nssm' # nssm | svc
$Drop = '^(MxGateway__Ldap__|MxGateway__Dashboard__GroupToRole__)'
$Add = @('MxGateway__Ldap__Enabled=true',
'MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com',
'MxGateway__Ldap__Port=636',
'MxGateway__Ldap__Transport=Ldaps',
'MxGateway__Ldap__AllowInsecure=false',
'MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636',
'MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com',
'MxGateway__Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com',
'MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}',
'MxGateway__Ldap__UserNameAttribute=sAMAccountName',
'MxGateway__Ldap__DisplayNameAttribute=name',
'MxGateway__Ldap__GroupAttribute=memberOf',
'MxGateway__Ldap__BindMode=ServiceAccount',
'MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator',
'MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_QA=Viewer',
'MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer')
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$Svc"
if ($Kind -eq 'nssm') { $k = "$k\Parameters"; $prop = 'AppEnvironmentExtra' } else { $prop = 'Environment' }
$before = @((Get-ItemProperty $k -Name $prop).$prop)
$kept = @($before | Where-Object { $n = $_.Substring(0, [Math]::Max(0,$_.IndexOf('='))); $n -notmatch $Drop })
$dropped = @($before | Where-Object { $kept -notcontains $_ } | ForEach-Object { $_.Substring(0,$_.IndexOf('=')) })
foreach ($a in $Add) { $n = $a.Substring(0,$a.IndexOf('=')); if (@($kept | Where-Object { $_ -like "$n=*" }).Count) { throw "add collides with kept entry $n" } }
$new = $kept + $Add
Set-ItemProperty -Path $k -Name $prop -Value ([string[]]$new) -Type MultiString
$after = @((Get-ItemProperty $k -Name $prop).$prop)
if ($after.Count -ne $new.Count) { throw 'count mismatch after write' }
for ($i = 0; $i -lt $new.Count; $i++) { if ($after[$i] -cne $new[$i]) { throw "entry $i differs after write" } }
"$Svc ($prop) $($before.Count) -> $($after.Count) entries; dropped: $($dropped -join ', ')"
foreach ($a in $Add) { " + $a" }
@@ -0,0 +1,37 @@
# Rewrite one service's env block (NSSM AppEnvironmentExtra or SCM Environment): drop entries whose NAME matches
# $Drop, append $Add. Every other entry stays byte-identical (checked). No restart. Rendered per call.
$ErrorActionPreference = 'Stop'
$Svc = 'OtOpcUaHost'; $Kind = 'svc' # nssm | svc
$Drop = '^(Security__Ldap__|Security__Cookie__RequireHttpsCookie$)'
$Add = @('Security__Ldap__DevStubMode=false',
'Security__Ldap__Enabled=true',
'Security__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com',
'Security__Ldap__Port=636',
'Security__Ldap__Transport=Ldaps',
'Security__Ldap__AllowInsecure=false',
'Security__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636',
'Security__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com',
'Security__Ldap__ServiceAccountDn=svc_igntadmin@nam.zimmer.com',
'Security__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}',
'Security__Ldap__UserNameAttribute=sAMAccountName',
'Security__Ldap__DisplayNameAttribute=name',
'Security__Ldap__GroupAttribute=memberOf',
'Security__Ldap__BindMode=ServiceAccount',
'Security__Ldap__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator',
'Security__Ldap__GroupToRole__UG_WW_APP_IDE_QA=Designer',
'Security__Ldap__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer',
'Security__Ldap__GroupToRole__UG_WW_HIST_ADMIN_QA=Viewer',
'Security__Cookie__RequireHttpsCookie=false')
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$Svc"
if ($Kind -eq 'nssm') { $k = "$k\Parameters"; $prop = 'AppEnvironmentExtra' } else { $prop = 'Environment' }
$before = @((Get-ItemProperty $k -Name $prop).$prop)
$kept = @($before | Where-Object { $n = $_.Substring(0, [Math]::Max(0,$_.IndexOf('='))); $n -notmatch $Drop })
$dropped = @($before | Where-Object { $kept -notcontains $_ } | ForEach-Object { $_.Substring(0,$_.IndexOf('=')) })
foreach ($a in $Add) { $n = $a.Substring(0,$a.IndexOf('=')); if (@($kept | Where-Object { $_ -like "$n=*" }).Count) { throw "add collides with kept entry $n" } }
$new = $kept + $Add
Set-ItemProperty -Path $k -Name $prop -Value ([string[]]$new) -Type MultiString
$after = @((Get-ItemProperty $k -Name $prop).$prop)
if ($after.Count -ne $new.Count) { throw 'count mismatch after write' }
for ($i = 0; $i -lt $new.Count; $i++) { if ($after[$i] -cne $new[$i]) { throw "entry $i differs after write" } }
"$Svc ($prop) $($before.Count) -> $($after.Count) entries; dropped: $($dropped -join ', ')"
foreach ($a in $Add) { " + $a" }
@@ -0,0 +1,24 @@
# MxAccessGateway QA 76a71bc8 -> e74918cc with the NAM base appsettings.json. Rendered: __MODE__ = dry | real.
# The repo upgrade script carries the live appsettings*.json into the new app\, so the NAM base file is put in place
# first (GwAdmin/GwReader removed; the LDAP settings themselves live in the NSSM env, written by env-mxgw.ps1).
$ErrorActionPreference = 'Stop'
$Mode = '__MODE__'
$Stage = 'E:\psrem-stage\mxgw-e74918cc'
$Zip = Join-Path $Stage 'mxgw-e74918cc.zip'
$want = 'F0999B75BEE196375AEB8E3E277A0E948C54A747FD5251D336461C9665D847F2'
$got = (Get-FileHash $Zip -Algorithm SHA256).Hash
if ($got -ne $want) { "ABORT zip hash $got"; exit 1 }
$exp = Join-Path $Stage 'expanded'
if (-not (Test-Path (Join-Path $exp 'Worker\MxGateway.Worker.exe'))) { Expand-Archive -Path $Zip -DestinationPath $exp -Force }
"staged Server = " + [Diagnostics.FileVersionInfo]::GetVersionInfo((Join-Path $exp 'Server\ZB.MOM.WW.MxGateway.Server.dll')).ProductVersion
"staged Worker = " + [Diagnostics.FileVersionInfo]::GetVersionInfo((Join-Path $exp 'Worker\MxGateway.Worker.exe')).ProductVersion
$live = 'E:\ApiInstall\MxAccessGateway\app\appsettings.json'
if ($Mode -eq 'real') {
$nam = Join-Path $Stage 'appsettings.json.nam'
Get-Content $nam -Raw | ConvertFrom-Json | Out-Null
Copy-Item $nam $live -Force
"live appsettings.json <- NAM base, sha256 $((Get-FileHash $live).Hash.Substring(0,16))"
}
$a = @{ ServerSource = (Join-Path $exp 'Server'); WorkerSource = (Join-Path $exp 'Worker'); NssmPath = 'C:\ProgramData\ZB\qa-build\tools\nssm.exe'; ExpectedSha = 'e74918cc'; Label = 'nam-e74918cc' }
if ($Mode -ne 'real') { $a.DryRun = $true }
& (Join-Path $Stage 'Upgrade-MxGatewayService.ps1') @a 2>&1 | Out-String -Width 300
@@ -0,0 +1,80 @@
# QA OtOpcUa upgrade c9640f5c -> cdf433a3, phase 1 on ONE node (service keeps running): protected backup dir, registry
# export, script + zip hashes, expand to app.new-cdf433a3 + deps check, env NAMES/SHA prefixes (+ no OpcUa__Redundancy* /
# PeerApplicationUris*), pre-window -Verify with the new installer, swap -WhatIf. On vq03 only (-DbBackup): ConfigDb
# COPY_ONLY+CHECKSUM+COMPRESSION backup + RESTORE VERIFYONLY + migration state. Secrets never printed.
param([switch]$DbBackup)
$ErrorActionPreference = 'Stop'
try {
$stageDir = 'E:\psrem-stage\otopcua-cdf433a3'
$backupDir = "$stageDir\backup"
$candidate = 'E:\ApiInstall\OtOpcUa\app.new-cdf433a3'
$svcKeyPath = 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost'
. "$stageDir\envdigest.ps1"
Write-Output "node=$env:COMPUTERNAME"
if (-not (Test-Path -LiteralPath $backupDir)) { New-Item -ItemType Directory -Force -Path $backupDir | Out-Null }
& icacls.exe $backupDir /inheritance:r /grant:r 'SYSTEM:(OI)(CI)(F)' 'BUILTIN\Administrators:(OI)(CI)(F)' | Out-Null
if ($LASTEXITCODE -ne 0) { throw "icacls backup dir failed $LASTEXITCODE" }
Write-Output "backup dir $backupDir (protected: SYSTEM + Administrators)"
& reg.exe export 'HKLM\SYSTEM\CurrentControlSet\Services\OtOpcUaHost' "$backupDir\OtOpcUaHost-pre-cdf433a3.reg" /y | Out-Null
if ($LASTEXITCODE -ne 0) { throw "reg export failed $LASTEXITCODE" }
Write-Output "reg export ok"
$expectScripts = @{ 'Install-ScriptWorkerIsolation.ps1' = 'BB31D2759D0DE29292C123D74860226ED37AF554F2C8FE6871312C7E81F7C05A';
'ScriptWorkerIsolation.psm1' = '4C1C9D0B7086D66C3977E013933018D7495C14DB56E53701633688B10AB9D32C';
'Invoke-OtOpcUaInstallSwap.ps1' = '10C07BD0F527BDBDACC2B94605FB410996F0B9446390F88A8DF5C725AB0C7C6F' }
foreach ($scriptName in $expectScripts.Keys) {
if ((Get-FileHash -LiteralPath "$stageDir\install\$scriptName" -Algorithm SHA256).Hash -ne $expectScripts[$scriptName]) { throw "script hash mismatch $scriptName" }
}
Write-Output "install scripts: 3/3 hashes match the pack"
$wantHash = ((Get-Content -LiteralPath "$stageDir\otopcua-cdf433a3.zip.sha256" -Raw).Trim() -split '\s+')[0]
$gotHash = (Get-FileHash -LiteralPath "$stageDir\otopcua-cdf433a3.zip" -Algorithm SHA256).Hash
if ($wantHash -ne $gotHash) { throw "zip hash mismatch want=$wantHash got=$gotHash" }
if (Test-Path -LiteralPath $candidate) { Remove-Item -LiteralPath $candidate -Recurse -Force }
Expand-Archive -LiteralPath "$stageDir\otopcua-cdf433a3.zip" -DestinationPath $candidate
$fileCount = (Get-ChildItem -LiteralPath $candidate -Recurse -File | Measure-Object).Count
Write-Output ("expanded files={0} (expect 1410) zip hash ok {1}" -f $fileCount, $gotHash.Substring(0,12))
if ($fileCount -ne 1410) { throw "file count mismatch" }
$verInfo = (Get-Item -LiteralPath "$candidate\OtOpcUa.Host.dll").VersionInfo.ProductVersion
Write-Output "candidate ProductVersion=$verInfo"
if ($verInfo -notlike '1.0.0+cdf433a3512353de*') { throw "unexpected ProductVersion" }
$depsText = Get-Content -LiteralPath "$candidate\OtOpcUa.Host.deps.json" -Raw
foreach ($pkg in 'ZB.MOM.WW.LocalDb/0.3.8','ZB.MOM.WW.LocalDb.Replication/0.3.8','ZB.MOM.WW.HistorianGateway.Client/1.1.0') {
if ($depsText -notmatch [regex]::Escape("`"$pkg`"")) { throw "deps.json lacks $pkg" }
}
Write-Output "deps.json: LocalDb + .Replication 0.3.8, HistorianGateway.Client 1.1.0"
$envLines = @((Get-ItemProperty $svcKeyPath).Environment)
Show-ZbEnvDigest $envLines 'pre'
$envTable = @{}
foreach ($envLine in $envLines) { $eqPos = $envLine.IndexOf('='); if ($eqPos -gt 0) { $envTable[$envLine.Substring(0,$eqPos)] = $envLine.Substring($eqPos+1) } }
foreach ($flagName in 'Scripting__Runner','Scripting__Runner__Seams__0','Scripting__Runner__Seams__1','Scripting__Runner__Seams__2','Scripting__Worker__Identity','Scripting__Worker__PeCacheDir','OtOpcUa__Telemetry__Exporter','OtOpcUa__Telemetry__OtlpEndpoint') {
if ($envTable.ContainsKey($flagName)) { Write-Output " flag $flagName=$($envTable[$flagName])" } else { Write-Output " flag $flagName=<absent>" }
}
$redundancyNames = @($envTable.Keys | Where-Object { $_ -like 'OpcUa__Redundancy*' -or $_ -like 'OpcUa__PeerApplicationUris*' })
Write-Output (" OpcUa__Redundancy*/PeerApplicationUris* entries: {0}" -f $redundancyNames.Count)
# QA runs Warm since 2026-11-13: redundancy keys are expected (recorded above, not refused)
$appsettingsHits = @(Get-ChildItem -LiteralPath 'E:\ApiInstall\OtOpcUa\app' -Filter 'appsettings*.json' | Select-String -Pattern 'Redundancy|PeerApplicationUris')
Write-Output (" appsettings*.json Redundancy/PeerApplicationUris hits: {0}" -f $appsettingsHits.Count)
if ($DbBackup) {
$connBuilder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder($envTable['ConnectionStrings__ConfigDb'])
$sqlcmdExe = (Get-Command sqlcmd.exe -ErrorAction SilentlyContinue).Source
if (-not $sqlcmdExe) { $sqlcmdExe = 'C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\SQLCMD.EXE' }
$env:SQLCMDPASSWORD = $connBuilder.Password
try {
& $sqlcmdExe -S $connBuilder.DataSource -U $connBuilder.UserID -C -I -b -d master -W -h -1 -Q "BACKUP DATABASE [OtOpcUaConfig] TO DISK = N'OtOpcUaConfig_pre-cdf433a3_20261119.bak' WITH COPY_ONLY, INIT, CHECKSUM, COMPRESSION; RESTORE VERIFYONLY FROM DISK = N'OtOpcUaConfig_pre-cdf433a3_20261119.bak' WITH CHECKSUM; SELECT 'db backup + verify ok';"
if ($LASTEXITCODE -ne 0) { throw "db backup failed $LASTEXITCODE" }
& $sqlcmdExe -S $connBuilder.DataSource -U $connBuilder.UserID -C -I -b -d $connBuilder.InitialCatalog -W -h -1 -Q "SET NOCOUNT ON; SELECT 'migrations=' + CAST(COUNT(*) AS varchar(10)) + ' last=' + MAX(MigrationId) FROM __EFMigrationsHistory; SELECT 'snapshot=' + CAST(snapshot_isolation_state_desc AS varchar(20)) FROM sys.databases WHERE name = DB_NAME(); SELECT TOP 1 'latest sealed=' + LOWER(CONVERT(varchar(36),DeploymentId)) + ' rev=' + RevisionHash FROM dbo.Deployment WHERE Status = 2 ORDER BY SealedAtUtc DESC;"
if ($LASTEXITCODE -ne 0) { throw "migration query failed $LASTEXITCODE" }
} finally { Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue; Remove-Variable connBuilder -ErrorAction SilentlyContinue }
}
Set-Location "$stageDir\install"
Write-Output '--- pre-window -Verify (new installer, service up)'
& powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\Install-ScriptWorkerIsolation.ps1 -Verify *>&1 | ForEach-Object { "$_" } | Where-Object { $_ -notmatch 'does not inherit the worker deny|grants nothing to Everyone|^\s*(@\{|-DenyPaths\.|to -DenyPaths\.|add it to -DenyPaths\.|RESTRICTED, or add)' }
$verifyExit = $LASTEXITCODE
Write-Output "pre-window verify exit=$verifyExit"
if ($verifyExit -ne 0) { throw "pre-window verify failed" }
Write-Output '--- swap -WhatIf (preflight only)'
& .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate $candidate -PreviousInstall E:\ApiInstall\OtOpcUa\app.bak.20261119-pre-cdf433a3 -Carry 'appsettings*.json','logs' -WhatIf *>&1 | ForEach-Object { "$_" }
Write-Output ("E: free {0} GB" -f [int]((Get-PSDrive E).Free/1GB))
Write-Output "service=$((Get-Service OtOpcUaHost).Status)"
exit 0
} catch { Write-Output "PREPARE FAILED: $($_.Exception.Message)"; exit 1 }
@@ -0,0 +1,27 @@
# QA: start OtOpcUaHost on ONE node, wait for /health/ready 200 (up to 180 s), full -Verify, env digest, live version.
$ErrorActionPreference = 'Stop'
function Out-ZbStamped { process { '{0:HH:mm:ss.fff}Z {1}' -f (Get-Date).ToUniversalTime(), "$_" } }
try {
$stageDir = 'E:\psrem-stage\otopcua-cdf433a3'
. "$stageDir\envdigest.ps1"
"node=$env:COMPUTERNAME" | Out-ZbStamped
$t0 = Get-Date
'start issued' | Out-ZbStamped
Start-Service OtOpcUaHost
$deadline = (Get-Date).AddSeconds(180)
$readyCode = 0
while ((Get-Date) -lt $deadline) {
if ((Get-Service OtOpcUaHost).Status -ne 'Running') { Start-Sleep -Milliseconds 500; continue }
try { $readyCode = (Invoke-WebRequest -UseBasicParsing -Uri 'http://localhost:9000/health/ready' -TimeoutSec 5).StatusCode } catch { $readyCode = 0 }
if ($readyCode -eq 200) { break }
Start-Sleep -Milliseconds 500
}
("service={0} ready={1} ({2:N1}s after start issued)" -f (Get-Service OtOpcUaHost).Status, $readyCode, ((Get-Date)-$t0).TotalSeconds) | Out-ZbStamped
Set-Location "$stageDir\install"
& powershell.exe -NoProfile -ExecutionPolicy Bypass -File .\Install-ScriptWorkerIsolation.ps1 -Verify *>&1 | ForEach-Object { "$_" } | Where-Object { $_ -notmatch 'does not inherit the worker deny|grants nothing to Everyone|^\s*(@\{|-DenyPaths\.|to -DenyPaths\.|add it to -DenyPaths\.|RESTRICTED, or add)' } | Where-Object { $_ -match 'VERIFY|MISSING|coverage' }
"post-start verify exit=$LASTEXITCODE" | Out-ZbStamped
Show-ZbEnvDigest @((Get-ItemProperty 'HKLM:\SYSTEM\CurrentControlSet\Services\OtOpcUaHost').Environment) 'post' | Select-Object -First 1
Write-Output ("live ProductVersion=" + (Get-Item 'E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll').VersionInfo.ProductVersion)
if ($readyCode -ne 200) { exit 1 }
exit 0
} catch { Write-Output "START FAILED: $($_.Exception.Message)"; exit 1 }
@@ -0,0 +1,33 @@
# QA window on ONE node, started with the service RUNNING (pack §4 QA steps 2/3): Invoke-OtOpcUaInstallSwap.ps1
# -Carry appsettings*.json,logs -NoStart runs -Scope Outside before its own stop, stops, swaps, -Scope InstallRoot
# reapply+verify, leaves it STOPPED. Then back up data\ + pki\ (stopped => consistent SQLite triplets). Lines UTC-stamped.
$ErrorActionPreference = 'Stop'
function Out-ZbStamped { process { '{0:HH:mm:ss.fff}Z {1}' -f (Get-Date).ToUniversalTime(), "$_" } }
try {
$stageDir = 'E:\psrem-stage\otopcua-cdf433a3'
$backupDir = "$stageDir\backup"
$candidate = 'E:\ApiInstall\OtOpcUa\app.new-cdf433a3'
$prevInstall = 'E:\ApiInstall\OtOpcUa\app.bak.20261119-pre-cdf433a3'
"node=$env:COMPUTERNAME" | Out-ZbStamped
if (-not (Test-Path -LiteralPath $candidate)) { throw "candidate missing" }
if (Test-Path -LiteralPath $prevInstall) { throw "prev path already exists: $prevInstall" }
if ((Get-Service OtOpcUaHost).Status -ne 'Running') { throw "service not running; the pack starts the swap with the service up" }
'swap invoked (service running)' | Out-ZbStamped
Set-Location "$stageDir\install"
& .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate $candidate -PreviousInstall $prevInstall `
-Carry 'appsettings*.json','logs' -NoStart *>&1 | ForEach-Object { "$_" } |
Where-Object { $_ -notmatch 'does not inherit the worker deny|grants nothing to Everyone|^\s*(@\{|-DenyPaths\.|to -DenyPaths\.|add it to -DenyPaths\.|RESTRICTED, or add)' } | Out-ZbStamped
("swap returned, service={0}" -f (Get-Service OtOpcUaHost).Status) | Out-ZbStamped
if ((Get-Service OtOpcUaHost).Status -ne 'Stopped') { throw "service not stopped after the -NoStart swap" }
if (-not (Test-Path -LiteralPath $prevInstall)) { throw "swap did not move the live folder" }
if (Get-Process -Name 'OtOpcUa.Host' -ErrorAction SilentlyContinue) { throw "OtOpcUa.Host still running" }
$workerProc = Get-Process -Name 'ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue
if ($workerProc) { Start-Sleep 3; if (Get-Process -Name 'ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue) { throw "ScriptWorker still running" } }
foreach ($dirName in 'data','pki') {
& robocopy.exe "E:\ApiInstall\OtOpcUa\$dirName" "$backupDir\$dirName-pre-cdf433a3" /E /COPY:DAT /R:1 /W:1 /NFL /NDL /NJH /NP | Out-Null
if ($LASTEXITCODE -ge 8) { throw "$dirName backup robocopy failed $LASTEXITCODE" }
}
"data + pki backup ok -> $backupDir" | Out-ZbStamped
("staged ProductVersion=" + (Get-Item 'E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll').VersionInfo.ProductVersion) | Out-ZbStamped
exit 0
} catch { Write-Output "SWAP FAILED: $($_.Exception.Message)"; Write-Output "service=$((Get-Service OtOpcUaHost).Status)"; exit 1 }
@@ -0,0 +1,50 @@
# QA window on ONE node, started with the service RUNNING (pack §4 QA steps 2/3): Invoke-OtOpcUaInstallSwap.ps1
# -Carry appsettings*.json,logs -NoStart runs -Scope Outside before its own stop, stops, swaps, -Scope InstallRoot
# reapply+verify, leaves it STOPPED. Then back up data\ + pki\ (stopped => consistent SQLite triplets). Lines UTC-stamped.
$ErrorActionPreference = 'Stop'
function Out-ZbStamped { process { '{0:HH:mm:ss.fff}Z {1}' -f (Get-Date).ToUniversalTime(), "$_" } }
try {
$stageDir = 'E:\psrem-stage\otopcua-cdf433a3'
$backupDir = "$stageDir\backup"
$candidate = 'E:\ApiInstall\OtOpcUa\app.new-cdf433a3'
$prevInstall = 'E:\ApiInstall\OtOpcUa\app.bak.20261119-pre-cdf433a3'
"node=$env:COMPUTERNAME" | Out-ZbStamped
if (-not (Test-Path -LiteralPath $candidate)) { throw "candidate missing" }
if (Test-Path -LiteralPath $prevInstall) { throw "prev path already exists: $prevInstall" }
if ((Get-Service OtOpcUaHost).Status -ne 'Running') { throw "service not running; the pack starts the swap with the service up" }
'swap invoked (service running)' | Out-ZbStamped
Set-Location "$stageDir\install"
$swapFilter = 'does not inherit the worker deny|grants nothing to Everyone|^\s*(@\{|-DenyPaths\.|to -DenyPaths\.|add it to -DenyPaths\.|RESTRICTED, or add)'
try {
& .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate $candidate -PreviousInstall $prevInstall `
-Carry 'appsettings*.json','logs' -NoStart *>&1 | ForEach-Object { "$_" } | Where-Object { $_ -notmatch $swapFilter } | Out-ZbStamped
} catch {
# vq04 2026-11-13: the move right after the stop hit "file in use" (host/worker still exiting after SCM said Stopped).
# Retry once only when nothing moved: service Stopped, app\ intact, candidate still staged, no PreviousInstall.
("first swap attempt failed: {0}" -f $_.Exception.Message) | Out-ZbStamped
if ((Get-Service OtOpcUaHost).Status -ne 'Stopped' -or (Test-Path -LiteralPath $prevInstall) -or -not (Test-Path -LiteralPath $candidate) -or
-not (Test-Path -LiteralPath 'E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll')) { throw "state not retryable: $($_.Exception.Message)" }
$waitUntil = (Get-Date).AddSeconds(30)
while ((Get-Date) -lt $waitUntil -and (Get-Process -Name 'OtOpcUa.Host','ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue)) { Start-Sleep -Milliseconds 500 }
$left = @(Get-Process -Name 'OtOpcUa.Host','ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue)
("host/worker processes left: {0}" -f (($left | ForEach-Object { "$($_.ProcessName)#$($_.Id)" }) -join ', ')) | Out-ZbStamped
foreach ($p in $left) { Stop-Process -Id $p.Id -Force }
Start-Sleep -Seconds 2
'retrying the swap (service stopped)' | Out-ZbStamped
& .\Invoke-OtOpcUaInstallSwap.ps1 -InstallRoot E:\ApiInstall\OtOpcUa\app -Candidate $candidate -PreviousInstall $prevInstall `
-Carry 'appsettings*.json','logs' -NoStart *>&1 | ForEach-Object { "$_" } | Where-Object { $_ -notmatch $swapFilter } | Out-ZbStamped
}
("swap returned, service={0}" -f (Get-Service OtOpcUaHost).Status) | Out-ZbStamped
if ((Get-Service OtOpcUaHost).Status -ne 'Stopped') { throw "service not stopped after the -NoStart swap" }
if (-not (Test-Path -LiteralPath $prevInstall)) { throw "swap did not move the live folder" }
if (Get-Process -Name 'OtOpcUa.Host' -ErrorAction SilentlyContinue) { throw "OtOpcUa.Host still running" }
$workerProc = Get-Process -Name 'ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue
if ($workerProc) { Start-Sleep 3; if (Get-Process -Name 'ZB.MOM.WW.OtOpcUa.ScriptWorker' -ErrorAction SilentlyContinue) { throw "ScriptWorker still running" } }
foreach ($dirName in 'data','pki') {
& robocopy.exe "E:\ApiInstall\OtOpcUa\$dirName" "$backupDir\$dirName-pre-cdf433a3" /E /COPY:DAT /R:1 /W:1 /NFL /NDL /NJH /NP | Out-Null
if ($LASTEXITCODE -ge 8) { throw "$dirName backup robocopy failed $LASTEXITCODE" }
}
"data + pki backup ok -> $backupDir" | Out-ZbStamped
("staged ProductVersion=" + (Get-Item 'E:\ApiInstall\OtOpcUa\app\OtOpcUa.Host.dll').VersionInfo.ProductVersion) | Out-ZbStamped
exit 0
} catch { Write-Output "SWAP FAILED: $($_.Exception.Message)"; Write-Output "service=$((Get-Service OtOpcUaHost).Status)"; exit 1 }
@@ -0,0 +1,33 @@
# NAM LDAP cutover, QA vq03/vq04 — Phase 0 preflight + backups (read-only except the backup folder). Secret values are never printed.
$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue'
$ts = Get-Date -Format 'yyyyMMddTHHmmss'
$bk = "E:\ApiInstall\_ldapcut-$ts"; New-Item -ItemType Directory -Force $bk | Out-Null
& icacls.exe $bk /inheritance:r /grant:r 'SYSTEM:(OI)(CI)F' 'BUILTIN\Administrators:(OI)(CI)F' | Out-Null
"box $env:COMPUTERNAME backup dir $bk"
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') { foreach ($p in 636,389) {
$c = New-Object Net.Sockets.TcpClient; $ok = $c.ConnectAsync($h,$p).Wait(4000); $c.Dispose(); "reach $h`:$p $ok" } }
$root = @(Get-ChildItem Cert:\LocalMachine\Root | Where-Object { $_.Subject -like '*zimmer-rootca*' })
"zimmer-rootca in LocalMachine\Root: $($root.Count) $(($root | % { $_.Thumbprint + ' exp ' + $_.NotAfter.ToString('yyyy-MM-dd') }) -join '; ')"
foreach ($h in 'uswap-vdc-nam1.nam.zimmer.com','uswap-vdc-nam2.nam.zimmer.com') {
try { $t = New-Object Net.Sockets.TcpClient($h,636); $script:err = $null
$s = New-Object Net.Security.SslStream($t.GetStream(), $false, { param($a,$b,$c,$e) $script:err = $e; $true })
$s.AuthenticateAsClient($h); "tls $h policyErrors=$script:err proto=$($s.SslProtocol)"; $s.Dispose(); $t.Dispose() } catch { "tls $h FAILED $($_.Exception.Message)" } }
$svcs = Get-CimInstance Win32_Service | Where-Object { $_.PathName -match 'ApiInstall\\(HistorianGateway|MxAccessGateway|MxGateway|OtOpcUa|ScadaBridge)' -or $_.Name -match 'HistorianGateway|MxGateway|MxAccess|OtOpcUa|ScadaBridge' }
$show = '(?i)ENVIRONMENT$|DisableLogin|AllowOutside|Cookie|__Ldap__(Enabled|DevStub|Server|Port|Transport|AllowInsecure|SearchBase|ServiceAccountDn|UserNameAttribute|DisplayNameAttribute|GroupAttribute|BindMode|FallbackServers|UseTls|RequiredGroup)|GroupToRole|Secrets__.*(SqlitePath|Source|FilePath|EnvVarName|Provider|Enabled|Schema|Table|Mode)$|RequiredGroup|Thumbprint|Redundancy|PeerApplicationUris|ApplicationUri|^ScadaBridge__Node|Cluster__Roles|Runner|ASPNETCORE_URLS'
foreach ($s in $svcs) {
"== service $($s.Name) state=$($s.State) path=$($s.PathName)"
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$($s.Name)" "$bk\reg-$($s.Name).reg" /y | Out-Null
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$($s.Name)"
$p = Get-ItemProperty "$k\Parameters" -ErrorAction SilentlyContinue
if ($p.AppDirectory) { " nssm AppDirectory=$($p.AppDirectory) Application=$($p.Application)" }
$lines = @((Get-ItemProperty $k -ErrorAction SilentlyContinue).Environment | % { "[svc] $_" }) + @($p.AppEnvironmentExtra | % { "[nssm] $_" }) | Where-Object { $_ }
foreach ($l in $lines) { $tag, $l = $l -split " ", 2; $i = $l.IndexOf('='); if ($i -lt 1) { continue }; $n = $l.Substring(0,$i)
if ($n -match $show -and $l -notmatch '(?i)password|key=|pepper|secret=') { " $tag env $l" } else { " $tag env $n=<hidden len $($l.Length-$i-1)>" } }
}
foreach ($p in 'HistorianGateway\app\appsettings*.json','MxAccessGateway\app\appsettings*.json','OtOpcUa\app\appsettings*.json','ScadaBridge\app\appsettings*.json') {
foreach ($f in Get-ChildItem "E:\ApiInstall\$p" -ErrorAction SilentlyContinue) { $d = "$bk\" + ($f.FullName.Substring(15) -replace '\\','_'); Copy-Item $f.FullName $d; "copied $($f.FullName) sha256 $((Get-FileHash $d).Hash.Substring(0,16))" } }
foreach ($d in 'HistorianGateway','MxAccessGateway','OtOpcUa','ScadaBridge') { foreach ($dll in Get-ChildItem "E:\ApiInstall\$d\app" -Filter '*.dll' -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^(ZB\.MOM\.WW\.(HistorianGateway\.Server|OtOpcUa\.Host|ScadaBridge\.Host)|MxGateway\.Server)\.dll$' }) { "ver $d $($dll.Name) $($dll.VersionInfo.ProductVersion)" } }
Get-ChildItem 'E:\ApiInstall' -Directory | % { "dir $($_.Name)" }
Get-ChildItem 'E:\ApiInstall\_tools' -Recurse -Filter '*.exe' -ErrorAction SilentlyContinue | % { "tool $($_.FullName) $($_.VersionInfo.ProductVersion)" }
foreach ($u in 'http://127.0.0.1:9000/health/active','http://127.0.0.1:8085/health/active','http://127.0.0.1:8084/health/active') { try { $r = Invoke-WebRequest -UseBasicParsing $u -TimeoutSec 5; "$u $($r.StatusCode) $($r.Content)" } catch { "$u $($_.Exception.Response.StatusCode.value__)" } }
"PREFLIGHT DONE $bk"
@@ -0,0 +1,131 @@
# ScadaBridge QA (vq03/vq04) 061432c8 -> 2e0ce81e (#144/#145, no migration) + NAM AD LDAP, one window. Rendered per call: ACTION / TS.
# prep | backup (vq03) | stop-site | stop-central | swap | start-central | start-site | verify | rollback-swap
# Derived from ScadaBridge deploy-prep/2026-11-15-143/qa.tmpl.ps1 (migrate dropped; the swap also installs the per-node NAM
# appsettings.Central.json and removes the GLAuth password env entry from the Central service).
$ErrorActionPreference = 'Stop'
$ProgressPreference = 'SilentlyContinue'
$Action = '__ACTION__'
$ts = '__TS__'
$sha = '2e0ce81e'
$fromSha = '061432c8'
$zipSha256 = '6C706B0A76B7E9EC7CA00E12CEA19C5D86814BEAF66B541F527CBFC3D6572E23'
$stageDir = "E:\psrem-stage\sb-$sha"
$stage = "$stageDir\unpacked"
$root = 'E:\ApiInstall\ScadaBridge'
$log = "$root\_DEPLOY-NOTES-$sha-$ts.txt"
$settings = 'appsettings.json','appsettings.Central.json','appsettings.Site.json'
$centralKey = 'HKLM:\SYSTEM\CurrentControlSet\Services\ScadaBridge-Central'
$namCentral = "$stageDir\appsettings.Central.json.nam-" + $env:COMPUTERNAME.Substring($env:COMPUTERNAME.Length-4).ToLower()
$bak = "F:\MSSQL\BACKUP\ScadaBridge_pre-$($sha)_$ts.bak"
function Say($m) { $line = "[{0:HH:mm:ss.fff}] {1} {2}" -f (Get-Date), $env:COMPUTERNAME, $m; $line; Add-Content -Path $log -Value $line }
function Is-Dropped($e) { $e -like 'ScadaBridge__Security__Ldap__ServiceAccountPassword=*' }
function Digest($entries) {
$h = [System.Security.Cryptography.SHA256]::Create(); $o = @()
foreach ($e in $entries) { $i = $e.IndexOf('='); if ($i -lt 1) { $o += '<unparsable>'; continue }
$v = $e.Substring($i+1); $x = -join ($h.ComputeHash([Text.Encoding]::UTF8.GetBytes($v)) | % { $_.ToString('x2') })
$o += ("{0} sha256:{1} len={2}" -f $e.Substring(0,$i), $x.Substring(0,12), $v.Length) }
$h.Dispose(); $o | Sort-Object
}
function Wait-Ready($url, $seconds) {
$deadline = (Get-Date).AddSeconds($seconds)
while ((Get-Date) -lt $deadline) { try { if ((Invoke-WebRequest -UseBasicParsing $url -TimeoutSec 5).StatusCode -eq 200) { return $true } } catch {}; Start-Sleep 2 }
return $false
}
function Stop-Svc($s) {
if ((Get-Service $s).Status -ne 'Stopped') { Say "stopping $s"; Stop-Service $s -Force; (Get-Service $s).WaitForStatus('Stopped','00:02:00') }
Say "$s stopped"
}
function Sql-Init {
$envC = @((Get-ItemProperty $centralKey).Environment)
$cs = ($envC | Where-Object { $_ -like 'ScadaBridge__Database__ConfigurationDb=*' }) -replace '^[^=]*=',''
$script:sb = New-Object System.Data.SqlClient.SqlConnectionStringBuilder $cs
$script:db = $sb['Database']; if (-not $db) { $script:db = $sb.InitialCatalog }
if ($db -ne 'ScadaBridge') { throw "unexpected catalog" }
$env:SQLCMDPASSWORD = $sb.Password
}
function Sql-Scalar($q) { $r = & sqlcmd -S $sb.DataSource -d $db -U $sb.UserID -C -h -1 -W -b -Q "SET NOCOUNT ON; $q"; if ($LASTEXITCODE -ne 0) { throw "sql failed" }; ($r | Where-Object { $_ -and $_.Trim() } | Select-Object -First 1).Trim() }
$schemaQ = "SELECT CONCAT((SELECT COUNT(*) FROM __EFMigrationsHistory),'|',(SELECT MAX(MigrationId) FROM __EFMigrationsHistory),'|',(SELECT COUNT_BIG(*) FROM dbo.AuditLog))"
switch ($Action) {
'prep' {
Say "PREP $fromSha -> $sha"
if ((Get-FileHash "$stageDir\scadabridge-$sha.zip" -Algorithm SHA256).Hash -ne $zipSha256) { throw "zip hash mismatch" }
if (-not (Test-Path $namCentral)) { throw "missing $namCentral" }
Get-Content $namCentral -Raw | ConvertFrom-Json | Out-Null
if (Test-Path $stage) { Remove-Item $stage -Recurse -Force }
Expand-Archive "$stageDir\scadabridge-$sha.zip" $stage
$sv = (Get-Item "$stage\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
$sc = (Get-Item "$stage\cli\scadabridge.exe").VersionInfo.ProductVersion
if ($sv -notlike "*+$sha*" -or $sc -notlike "*+$sha*") { throw "staged exe $sv / $sc" }
$lv = (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion
if ($lv -notlike "*+$fromSha*") { throw "live exe $lv" }
Say "staged $sv; live $lv; app files $((Get-ChildItem "$stage\app" -Recurse -File).Count), cli $((Get-ChildItem "$stage\cli" -Recurse -File).Count)"
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') {
& reg.exe export "HKLM\SYSTEM\CurrentControlSet\Services\$s" "$stageDir\_envbak-$s-$ts.reg" /y | Out-Null
if ($LASTEXITCODE -ne 0) { throw "reg export $s failed" }
$e = @((Get-ItemProperty "HKLM:\SYSTEM\CurrentControlSet\Services\$s").Environment)
Say ("$s env entries {0}" -f $e.Count); Digest $e | % { Say " env $s $_" }
}
$keep = New-Item -ItemType Directory -Force -Path "$root\_appsettings-keep-$ts"
foreach ($f in $settings) { Copy-Item "$root\app\$f" "$keep\$f"; Say ("keep {0} sha256 {1}" -f $f, (Get-FileHash "$keep\$f" -Algorithm SHA256).Hash) }
}
'backup' {
Sql-Init
$pre = Sql-Scalar $schemaQ; Say "schema before: $pre"
if (-not $pre.StartsWith('65|')) { throw "expected 65 migrations" }
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "BACKUP DATABASE [$db] TO DISK = N'$bak' WITH COPY_ONLY, CHECKSUM, COMPRESSION, INIT" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "backup failed" }
& sqlcmd -S $sb.DataSource -d master -U $sb.UserID -C -b -Q "RESTORE VERIFYONLY FROM DISK = N'$bak' WITH CHECKSUM" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "verifyonly failed" }
Say "backup $bak VERIFYONLY ok"
}
'stop-site' { Stop-Svc 'ScadaBridge-Site' }
'stop-central' {
Stop-Svc 'ScadaBridge-Central'
$deadline = (Get-Date).AddSeconds(30)
while ((Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) -and (Get-Date) -lt $deadline) { Start-Sleep 1 }
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe still running" }
Say "no Host.exe remains"
}
'swap' {
if (Get-Process -Name 'ZB.MOM.WW.ScadaBridge.Host' -ErrorAction SilentlyContinue) { throw "Host.exe running" }
Rename-Item "$root\app" "app.bak.$ts"; Rename-Item "$root\cli" "cli.bak.$ts"
Say "renamed app/cli -> *.bak.$ts"
Copy-Item "$stage\app" "$root\app" -Recurse
Copy-Item "$stage\cli" "$root\cli" -Recurse
$keep = "$root\_appsettings-keep-$ts"
foreach ($f in $settings) {
Copy-Item "$keep\$f" "$root\app\$f" -Force
if ((Get-FileHash "$keep\$f" -Algorithm SHA256).Hash -ne (Get-FileHash "$root\app\$f" -Algorithm SHA256).Hash) { throw "appsettings mismatch $f" }
Say "restored $f (sha256 match)"
}
Copy-Item $namCentral "$root\app\appsettings.Central.json" -Force
Say ("installed NAM appsettings.Central.json sha256 {0}" -f (Get-FileHash "$root\app\appsettings.Central.json" -Algorithm SHA256).Hash)
Say ("live Host " + (Get-Item "$root\app\ZB.MOM.WW.ScadaBridge.Host.exe").VersionInfo.ProductVersion + " / CLI " + (Get-Item "$root\cli\scadabridge.exe").VersionInfo.ProductVersion)
$before = @((Get-ItemProperty $centralKey).Environment)
$keepEnv = @($before | Where-Object { -not (Is-Dropped $_) })
Set-ItemProperty -Path $centralKey -Name Environment -Value ([string[]]$keepEnv) -Type MultiString
$after = @((Get-ItemProperty $centralKey).Environment)
foreach ($e in $keepEnv) { if ($after -notcontains $e) { throw "env entry changed" } }
if (@($after | ? { Is-Dropped $_ }).Count -ne 0) { throw "GLAuth password entry still present" }
Say ("central env entries {0} -> {1}" -f $before.Count, $after.Count)
}
'start-central' { Say "starting ScadaBridge-Central"; Start-Service 'ScadaBridge-Central'; $ok = Wait-Ready 'http://localhost:8085/health/ready' 180; Say "central ready=$ok"; if (-not $ok) { throw "central not ready" } }
'start-site' { Say "starting ScadaBridge-Site"; Start-Service 'ScadaBridge-Site'; $ok = Wait-Ready 'http://localhost:8084/health/ready' 180; Say "site ready=$ok"; if (-not $ok) { throw "site not ready" } }
'verify' {
foreach ($s in 'ScadaBridge-Central','ScadaBridge-Site') { Say ("$s " + (Get-Service $s).Status) }
if ($env:COMPUTERNAME -eq 'WONDER-APP-VQ03') { Sql-Init; Say ("schema now: " + (Sql-Scalar $schemaQ)) }
}
'rollback-swap' {
foreach ($s in 'ScadaBridge-Site','ScadaBridge-Central') { Stop-Svc $s }
if (Test-Path "$root\app") { Rename-Item "$root\app" "app.failed.$ts" }
if (Test-Path "$root\cli") { Rename-Item "$root\cli" "cli.failed.$ts" }
Rename-Item "$root\app.bak.$ts" 'app'; Rename-Item "$root\cli.bak.$ts" 'cli'
& reg.exe import "$stageDir\_envbak-ScadaBridge-Central-$ts.reg" | Out-Null
Say "rolled back binaries + Central env (services left STOPPED; start Central vq03, vq04, then Sites)"
}
default { throw "bad action $Action" }
}
Remove-Item Env:SQLCMDPASSWORD -ErrorAction SilentlyContinue
Say "$Action OK"
exit 0
@@ -0,0 +1,46 @@
# Seed nam-ldap-svc-password into every QA app store on THIS box, reading each store's location and KEK from the
# service's own env (never assumed). ScadaBridge Central's store is the shared SQL Server store: seeded once, on vq03,
# with a CLI variant that honours Secrets:Store:Provider=SqlServer. The value arrives in $vf over SMB and is deleted
# at the end; it is never printed. Usage: rendered with __BK__ (the box's protected _ldapcut dir).
$ErrorActionPreference = 'Stop'
$bk = '__BK__'
$vf = "$bk\_v.tmp"
$cli = 'E:\ApiInstall\_tools\secret-0.6.6\ZB.MOM.WW.Secrets.Cli.exe'
$cliSql = 'E:\ApiInstall\_tools\secret-0.6.6-sql\ZB.MOM.WW.Secrets.Cli.exe'
$name = 'nam-ldap-svc-password'
$desc = 'NAM LDAP lookup account (svc_igntadmin, QA interim 2026-11-19)'
& icacls.exe $vf /inheritance:r /grant:r 'BUILTIN\Administrators:F' 'NT AUTHORITY\SYSTEM:F' | Out-Null
function SvcEnv($svc) {
$k = "HKLM:\SYSTEM\CurrentControlSet\Services\$svc"
$l = @((Get-ItemProperty $k -ErrorAction SilentlyContinue).Environment) + @((Get-ItemProperty "$k\Parameters" -ErrorAction SilentlyContinue).AppEnvironmentExtra) | Where-Object { $_ }
$h = @{}; foreach ($e in $l) { $i = $e.IndexOf('='); if ($i -gt 0) { $h[$e.Substring(0,$i)] = $e.Substring($i+1) } }; $h
}
$targets = @('HistorianGateway','MxAccessGw','OtOpcUaHost')
if ($env:COMPUTERNAME -eq 'WONDER-APP-VQ03') { $targets += 'ScadaBridge-Central' }
$names = 'Secrets__SqlitePath','Secrets__MasterKey__Source','Secrets__MasterKey__FilePath','Secrets__MasterKey__EnvVarName','Secrets__Store__Provider','Secrets__SqlServer__ConnectionString','ZB_SECRETS_MASTER_KEY'
$fail = 0
try {
$want = [IO.File]::ReadAllText($vf)
"value file read, length $($want.Length)"
foreach ($svc in $targets) {
$e = SvcEnv $svc
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue }
foreach ($n in $names) { if ($e.ContainsKey($n)) { Set-Item "Env:$n" $e[$n] } }
$exe = $cli
if ($svc -eq 'ScadaBridge-Central') { $env:Secrets__Store__Provider = 'SqlServer'; $exe = $cliSql
if (-not $env:Secrets__SqlServer__ConnectionString) { throw 'Central env has no Secrets__SqlServer__ConnectionString' } }
if ($env:Secrets__MasterKey__Source -ne 'File') { "$svc master key source '$($env:Secrets__MasterKey__Source)' - not File, skipping"; $fail++; continue }
$where = if ($svc -eq 'ScadaBridge-Central') { 'SqlServer shared store' } else { $env:Secrets__SqlitePath }
$out = & cmd.exe /c "`"$exe`" set $name - --description `"$desc`" < `"$vf`"" 2>&1
$rc = $LASTEXITCODE
$got = (& $exe get $name 2>$null | Out-String).TrimEnd("`r","`n")
$match = ($got -ceq $want)
if (-not $match) { try { $j = $got | ConvertFrom-Json; $match = ($j.value -ceq $want) } catch {} }
"$svc -> $where kek=$($env:Secrets__MasterKey__FilePath) set rc=$rc out=$(($out | Out-String).Trim()) readback-match=$match"
if ($rc -ne 0 -or -not $match) { $fail++ }
}
} finally {
Remove-Item $vf -Force -ErrorAction SilentlyContinue; "value file removed: $(-not (Test-Path $vf))"
foreach ($n in $names) { Remove-Item "Env:$n" -ErrorAction SilentlyContinue }
}
"SEED DONE failures=$fail"; exit $fail
+5 -2
View File
@@ -1,4 +1,4 @@
# NAM AD LDAP cutover — DEV vd03, then QA vq03/vq04 (plan, NOT executed)
# NAM AD LDAP cutover — DEV vd03, then QA vq03/vq04 (DEV + QA EXECUTED 2026-11-19 on the interim account)
> **Status lives in Gitea.** Execution is tracked by
> [scadaproj#6](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/6). The dedicated, shared LDAP-lookup service
@@ -119,7 +119,10 @@ warning, not the second. Ask IT for the account's lockout state before any retry
---
## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19, NOT executed)
## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19; EXECUTED 2026-11-19)
**Executed 2026-11-19 15:29–16:10 EST:** all four apps on both boxes on the DEV builds and NAM AD; record
[`docs/deploy/2026-11-19-qa-nam-ldap/`](../deploy/2026-11-19-qa-nam-ldap/README.md). Optional step 8 (oswaltk) not run.
**Owner decisions (2026-11-19):**
1. QA binds as **`svc_igntadmin`** as well (interim). This supersedes A6 and the §7 note "QA never runs on