Files
scadaproj/docs/deploy/2026-11-19-qa-nam-ldap/README.md
T

9.9 KiB
Raw Blame History

2026-11-19: QA vq03/vq04 NAM AD LDAP cutover + DEV builds (HistorianGateway, MxAccessGateway, ScadaBridge, OtOpcUa)

Owner go (2026-11-19): "do the qa ldap plan, winrm usage approved". Plan: docs/plans/2026-11-15-nam-ldap-cutover.md §0Q (svc_igntadmin interim, DEV builds, one window). DEV record: ../2026-11-19-vd03-nam-ldap/. Umbrella: scadaproj#6.

Result: all four apps on both QA boxes now run the DEV builds and sign users in against NAM AD over LDAPS (nam1, nam2 as fallback), bound as svc_igntadmin@nam.zimmer.com with the password from ${secret:nam-ldap-svc-password} in each app's own store. The GLAuth multi-role identity no longer signs in anywhere on QA. Zero service-account bind failures and zero LDAP warnings in either box's OTel log export for the whole window.

App Build Where the LDAP config now lives
HistorianGateway (per box) 6365f34 → f84af29 NSSM env Ldap__* + Dashboard__GroupToRole__*
MxAccessGateway (per box) 76a71bc8 → e74918cc NSSM env MxGateway__Ldap__* + MxGateway__Dashboard__GroupToRole__*; base app\appsettings.json lost GwAdmin/GwReader
ScadaBridge (pair) 061432c8 → 2e0ce81e each Central's app\appsettings.Central.json (ScadaBridge:Security:Ldap); roles in LdapGroupMappings
OtOpcUa (Warm pair) 76e73e7d → cdf433a3 service env Security__Ldap__* (+ Security__Cookie__RequireHttpsCookie=false)

Every artifact is the one DEV runs, pulled from vd03's stage dirs over SMB and re-hashed (HG zip 1595a828…, mxgw f0999b75…, OtOpcUa c89ba972…, ScadaBridge 6c706b0a…). No app had a schema change.

Steps (EST)

Time Step
15:29 Phase 0 (preflight.ps1, both boxes): nam1/nam2 reachable on 636 + 389; zimmer-rootca in LocalMachine\Root (78D941D7…, exp 2036); LDAPS chains validate (TLS 1.3, no policy errors). Every service key and appsettings*.json → E:\ApiInstall\_ldapcut-20261119T152947 (vq03) / …T152958 (vq04), protected DACL. (The first run died on a parsing bug after one reg export; its …T152920/…T152930 dirs hold only that file.) OtOpcUa Primary vq04, ScadaBridge active central vq03. HG and mxgw already had RequireHttpsCookie=false in env; OtOpcUa had no cookie setting
15:35 Staged on both boxes: the four builds under E:\psrem-stage\{hg-f84af29,mxgw-e74918cc,otopcua-cdf433a3,sb-2e0ce81e}, the Secrets CLI 0.6.6 at E:\ApiInstall\_tools\secret-0.6.6\. Hashes checked on box
15:37 ScadaBridge role rows by CLI as GLAuth multi-role against the active central (audited path): ids 6–10, UG_WW_APP_ADMIN_QA→Administrator, UG_WW_APP_IDE_QA→Designer, UG_WW_APP_IDE_RO_QA→Viewer, UG_WW_HIST_ADMIN_QA→Deployer (interim), UG_WW_BTMGR_RO_QA→Viewer
15:41 Secret seeded (seed-secret.ps1; each store's path and KEK read from its own service env): HG, mxgw, OtOpcUa on each box (SQLite, KEK files under C:\ProgramData\ZB\kek\), ScadaBridge once, into the shared SQL Server store (zbsecrets schema; now 1 row). Read-back matched in all 7; the value file was ACL'd and deleted. Both Centrals' scadabridge.kek are identical (A3BD293A…). See finding 2 for the SQL Server seeding
15:43–15:48 HistorianGateway vq03 then vq04: env-hg.ps1 (34 → 41 entries: NAM Ldap__*, QA GroupToRole; SCADA-* dropped), then the f84af29 pack (preflight.ps1 0 FAIL / 0 WARN incl. a side-by-side boot with the new env, swap.ps1, verify.ps1 0 FAIL). Downtime 6.6 s / 7.7 s
15:56–15:59 MxAccessGateway vq03 then vq04: env-mxgw.ps1 (33 → 38: NAM MxGateway__Ldap__*, QA GroupToRole; SCADA-* dropped), then mxgw-upgrade.tmpl.ps1 (puts the NAM base appsettings.json in place, then the repo upgrade script with the START_PENDING try/catch patch from vd03; it carries the live appsettings into the new app\). Dry run first on vq03. Env fingerprint unchanged through stop/swap/start/60 s stability on both
16:00 ScadaBridge prep both nodes; DB backup F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak (VERIFYONLY ok; 65 migrations)
16:00:53–16:04:14 ScadaBridge window (sb-qa.tmpl.ps1): stop Site vq03, vq04, Central vq03, vq04; swap both (binaries + each node's own NAM appsettings.Central.json; ScadaBridge__Security__Ldap__ServiceAccountPassword removed, 15 → 14 env entries); start Central vq03 (16:03:17), vq04 (16:03:37), Site vq03 (16:03:55), vq04 (16:04:14). Site data path ~2 min 17 s, central API ~55 s
16:05–16:10 OtOpcUa (otopcua-prepare/swap2/start.ps1, the 11-13 QA pack retargeted): prepare both (1410 files, deps ok, ConfigDb OtOpcUaConfig_pre-cdf433a3_20261119.bak + VERIFYONLY on vq03, 9 migrations, isolation -Verify OK); swap the Standby vq03 (-NoStart), env-otopcua.ps1 (51 → 61); swap vq04 (stop 16:08:33), env (50 → 60); start vq04 (ready 16:09:29, 37.8 s), vq03 (ready 16:09:49, 8 s). Full isolation -Verify OK 21/21 on both. Pair outage ~56 s

Mappings now on QA

Group OtOpcUa ScadaBridge MxAccessGateway HistorianGateway
UG_WW_APP_ADMIN_QA Administrator Administrator Administrator Administrator
UG_WW_APP_IDE_QA Designer Designer Viewer —
UG_WW_APP_IDE_RO_QA Viewer Viewer Viewer Viewer
UG_WW_HIST_ADMIN_QA Viewer Deployer (interim) — Administrator
UG_WW_BTMGR_RO_QA — Viewer — Viewer

ScadaBridge still holds the five SCADA-* seed rows (ids 1–5); they grant nothing in NAM.

Verification

Check HG :5220 mxgw :5130 ScadaBridge :8085 OtOpcUa :9000
Anonymous GET / → login 302 both 302 both 302 302 both
Bogus user refused ✓ both ✓ both ✓ (GLAuth multi-role refused too, UI + CLI) ✓ both
dohertj2 signs in ✓ both ✓ both ✓ cookie; /auth/token 200 roles Administrator+Designer+Deployer; CLI (HTTP Basic) instance list (11) + admin role-mapping list ✓ cookie, GET / 200 (no redirect loop); /auth/token 200 roles Viewer+Designer+Administrator
After pack verify.ps1 0 FAIL both; ReadAggregate gate as expected /health/ready 200, env unchanged central + site ready on both; active central vq03; site-a both nodes online; MxGateway Connected 2196/2196 Good; WarmCanary Connected (to the Primary, vq04) 2/2 Good after the OtOpcUa restart; LocalDb replication connected vq04 Primary 250 (warm_role 1), vq03 Standby 90; localdb_oplog_depth 0 both; script workers up (shards 0, 1) both

Overview :5320 untouched. Not done, by owner instruction: DC-failover checks and firewall rules. Not done: the optional oswaltk Viewer-only login (step 8; needs that user).

Findings

  1. HG pack assumed QA was on fa9cf31. QA has run 6365f34 since 11-13. The pack's QA source SHA was patched (preflight/swap) before the run; nothing else in the pack needed changing.
  2. The headless Secrets CLI cannot write the SQL Server store. secret set registers only AddZbSecrets (SQLite); the interactive console does honour Secrets:SqlServer, but needs a TTY. ScadaBridge's Central uses Secrets:Store:Provider=SqlServer, so a one-off CLI build (0.6.6 + "if Secrets:Store:Provider=SqlServer, call AddZbSecretsSqlServerStore", mirroring ScadaBridge's own registration) seeded it, staged at E:\ApiInstall\_tools\secret-0.6.6-sql\ on vq03. Follow-up: make the headless CLI honour the provider key the same way, in ZB.MOM.WW.Secrets.
  3. mxgw on QA keeps its LDAP settings in the NSSM env, not in appsettings.json (unlike vd03). The base file's GwAdmin/GwReader merge with the env GroupToRole, so they were still removed from the file. The repo upgrade script carries the live appsettings*.json forward, so future redeploys keep the edit.
  4. OtOpcUa's first start after a pair stop took 37.8 s (vq04, alone in the mesh); the pair outage was ~56 s against ~37 s on 11-13. The swap itself had no file in use retry this time (the #908 fix is in the cdf433a3 installer).
  5. Noise seen, not caused here: OtOpcUa "Mesh PSK secret ot-mesh-psk-dev for cluster DEV is absent" (QA has no DEV cluster mesh peer); HG plaintext/relative-path production-readiness warnings.

Rollback (per app)

Keys and files: E:\ApiInstall\_ldapcut-20261119T152947 (vq03) / …T152958 (vq04) (reg-<service>.reg, <app>_app_appsettings*.json).

  • HG: the pack's rollback.ps1 (→ app.bak.20261119T154443 vq03 / …T154740 vq04), then reg import reg-HistorianGateway.reg, restart.
  • mxgw: Upgrade-MxGatewayService.ps1 -Rollback (stamps 20261119-155641 vq03 / 20261119-155819 vq04), then reg import reg-MxAccessGw.reg and restore MxAccessGateway_app_appsettings.json, restart.
  • ScadaBridge: rollback-swap action (app.bak.20261119T1600, Central env from _envbak-ScadaBridge-Central-20261119T1600.reg), start Central vq03, vq04, then the Sites. No DB restore needed. Keep the UG_WW_* rows.
  • OtOpcUa: stop both; swap app.bak.20261119-pre-cdf433a3 back; reg import reg-OtOpcUaHost.reg on each; start both.

The seeded secrets are inert once nothing references them. The box-local GLAuth (ScadaBridge-LDAP) is still running until the soak ends.

Automation identity

QA automation now uses NAM\dohertj2 (HTTP Basic / --password-file; Keychain winrm-cli/vq03). No checked-in QA script used multi-role; the remaining multi-role users are the local Docker rig scripts, which stay on GLAuth.

Files

preflight.ps1, seed-secret.ps1, env-edit.ps1 (+ rendered env-hg.ps1, env-mxgw.ps1, env-otopcua.ps1), mxgw-upgrade.tmpl.ps1, sb-qa.tmpl.ps1, otopcua-prepare.ps1, otopcua-swap.ps1, otopcua-swap2.ps1, otopcua-start.ps1. Login probes reused ../2026-11-19-vd03-nam-ldap/login-test.sh, sb-cli.sh, otelgrep.py.