Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
2026-11-19: QA vq03/vq04 NAM AD LDAP cutover + DEV builds (HistorianGateway, MxAccessGateway, ScadaBridge, OtOpcUa)
Owner go (2026-11-19): "do the qa ldap plan, winrm usage approved". Plan:
docs/plans/2026-11-15-nam-ldap-cutover.md §0Q (svc_igntadmin interim, DEV builds,
one window). DEV record: ../2026-11-19-vd03-nam-ldap/. Umbrella: scadaproj#6.
Result: all four apps on both QA boxes now run the DEV builds and sign users in against NAM AD over LDAPS (nam1, nam2
as fallback), bound as svc_igntadmin@nam.zimmer.com with the password from ${secret:nam-ldap-svc-password} in each
app's own store. The GLAuth multi-role identity no longer signs in anywhere on QA. Zero service-account bind
failures and zero LDAP warnings in either box's OTel log export for the whole window.
| App | Build | Where the LDAP config now lives |
|---|---|---|
| HistorianGateway (per box) | 6365f34 → f84af29 |
NSSM env Ldap__* + Dashboard__GroupToRole__* |
| MxAccessGateway (per box) | 76a71bc8 → e74918cc |
NSSM env MxGateway__Ldap__* + MxGateway__Dashboard__GroupToRole__*; base app\appsettings.json lost GwAdmin/GwReader |
| ScadaBridge (pair) | 061432c8 → 2e0ce81e |
each Central's app\appsettings.Central.json (ScadaBridge:Security:Ldap); roles in LdapGroupMappings |
| OtOpcUa (Warm pair) | 76e73e7d → cdf433a3 |
service env Security__Ldap__* (+ Security__Cookie__RequireHttpsCookie=false) |
Every artifact is the one DEV runs, pulled from vd03's stage dirs over SMB and re-hashed (HG zip 1595a828…, mxgw
f0999b75…, OtOpcUa c89ba972…, ScadaBridge 6c706b0a…). No app had a schema change.
Steps (EST)
| Time | Step |
|---|---|
| 15:29 | Phase 0 (preflight.ps1, both boxes): nam1/nam2 reachable on 636 + 389; zimmer-rootca in LocalMachine\Root (78D941D7…, exp 2036); LDAPS chains validate (TLS 1.3, no policy errors). Every service key and appsettings*.json → E:\ApiInstall\_ldapcut-20261119T152947 (vq03) / …T152958 (vq04), protected DACL. (The first run died on a parsing bug after one reg export; its …T152920/…T152930 dirs hold only that file.) OtOpcUa Primary vq04, ScadaBridge active central vq03. HG and mxgw already had RequireHttpsCookie=false in env; OtOpcUa had no cookie setting |
| 15:35 | Staged on both boxes: the four builds under E:\psrem-stage\{hg-f84af29,mxgw-e74918cc,otopcua-cdf433a3,sb-2e0ce81e}, the Secrets CLI 0.6.6 at E:\ApiInstall\_tools\secret-0.6.6\. Hashes checked on box |
| 15:37 | ScadaBridge role rows by CLI as GLAuth multi-role against the active central (audited path): ids 6–10, UG_WW_APP_ADMIN_QA→Administrator, UG_WW_APP_IDE_QA→Designer, UG_WW_APP_IDE_RO_QA→Viewer, UG_WW_HIST_ADMIN_QA→Deployer (interim), UG_WW_BTMGR_RO_QA→Viewer |
| 15:41 | Secret seeded (seed-secret.ps1; each store's path and KEK read from its own service env): HG, mxgw, OtOpcUa on each box (SQLite, KEK files under C:\ProgramData\ZB\kek\), ScadaBridge once, into the shared SQL Server store (zbsecrets schema; now 1 row). Read-back matched in all 7; the value file was ACL'd and deleted. Both Centrals' scadabridge.kek are identical (A3BD293A…). See finding 2 for the SQL Server seeding |
| 15:43–15:48 | HistorianGateway vq03 then vq04: env-hg.ps1 (34 → 41 entries: NAM Ldap__*, QA GroupToRole; SCADA-* dropped), then the f84af29 pack (preflight.ps1 0 FAIL / 0 WARN incl. a side-by-side boot with the new env, swap.ps1, verify.ps1 0 FAIL). Downtime 6.6 s / 7.7 s |
| 15:56–15:59 | MxAccessGateway vq03 then vq04: env-mxgw.ps1 (33 → 38: NAM MxGateway__Ldap__*, QA GroupToRole; SCADA-* dropped), then mxgw-upgrade.tmpl.ps1 (puts the NAM base appsettings.json in place, then the repo upgrade script with the START_PENDING try/catch patch from vd03; it carries the live appsettings into the new app\). Dry run first on vq03. Env fingerprint unchanged through stop/swap/start/60 s stability on both |
| 16:00 | ScadaBridge prep both nodes; DB backup F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak (VERIFYONLY ok; 65 migrations) |
| 16:00:53–16:04:14 | ScadaBridge window (sb-qa.tmpl.ps1): stop Site vq03, vq04, Central vq03, vq04; swap both (binaries + each node's own NAM appsettings.Central.json; ScadaBridge__Security__Ldap__ServiceAccountPassword removed, 15 → 14 env entries); start Central vq03 (16:03:17), vq04 (16:03:37), Site vq03 (16:03:55), vq04 (16:04:14). Site data path ~2 min 17 s, central API ~55 s |
| 16:05–16:10 | OtOpcUa (otopcua-prepare/swap2/start.ps1, the 11-13 QA pack retargeted): prepare both (1410 files, deps ok, ConfigDb OtOpcUaConfig_pre-cdf433a3_20261119.bak + VERIFYONLY on vq03, 9 migrations, isolation -Verify OK); swap the Standby vq03 (-NoStart), env-otopcua.ps1 (51 → 61); swap vq04 (stop 16:08:33), env (50 → 60); start vq04 (ready 16:09:29, 37.8 s), vq03 (ready 16:09:49, 8 s). Full isolation -Verify OK 21/21 on both. Pair outage ~56 s |
Mappings now on QA
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
UG_WW_APP_ADMIN_QA |
Administrator | Administrator | Administrator | Administrator |
UG_WW_APP_IDE_QA |
Designer | Designer | Viewer | — |
UG_WW_APP_IDE_RO_QA |
Viewer | Viewer | Viewer | Viewer |
UG_WW_HIST_ADMIN_QA |
Viewer | Deployer (interim) | — | Administrator |
UG_WW_BTMGR_RO_QA |
— | Viewer | — | Viewer |
ScadaBridge still holds the five SCADA-* seed rows (ids 1–5); they grant nothing in NAM.
Verification
| Check | HG :5220 |
mxgw :5130 |
ScadaBridge :8085 |
OtOpcUa :9000 |
|---|---|---|---|---|
Anonymous GET / → login |
302 both | 302 both | 302 | 302 both |
| Bogus user refused | ✓ both | ✓ both | ✓ (GLAuth multi-role refused too, UI + CLI) |
✓ both |
dohertj2 signs in |
✓ both | ✓ both | ✓ cookie; /auth/token 200 roles Administrator+Designer+Deployer; CLI (HTTP Basic) instance list (11) + admin role-mapping list |
✓ cookie, GET / 200 (no redirect loop); /auth/token 200 roles Viewer+Designer+Administrator |
| After | pack verify.ps1 0 FAIL both; ReadAggregate gate as expected |
/health/ready 200, env unchanged |
central + site ready on both; active central vq03; site-a both nodes online; MxGateway Connected 2196/2196 Good; WarmCanary Connected (to the Primary, vq04) 2/2 Good after the OtOpcUa restart; LocalDb replication connected | vq04 Primary 250 (warm_role 1), vq03 Standby 90; localdb_oplog_depth 0 both; script workers up (shards 0, 1) both |
Overview :5320 untouched. Not done, by owner instruction: DC-failover checks and firewall rules. Not done: the
optional oswaltk Viewer-only login (step 8; needs that user).
Findings
- HG pack assumed QA was on
fa9cf31. QA has run6365f34since 11-13. The pack's QA source SHA was patched (preflight/swap) before the run; nothing else in the pack needed changing. - The headless Secrets CLI cannot write the SQL Server store.
secret setregisters onlyAddZbSecrets(SQLite); the interactive console does honourSecrets:SqlServer, but needs a TTY. ScadaBridge's Central usesSecrets:Store:Provider=SqlServer, so a one-off CLI build (0.6.6 + "ifSecrets:Store:Provider=SqlServer, callAddZbSecretsSqlServerStore", mirroring ScadaBridge's own registration) seeded it, staged atE:\ApiInstall\_tools\secret-0.6.6-sql\on vq03. Follow-up: make the headless CLI honour the provider key the same way, inZB.MOM.WW.Secrets. - mxgw on QA keeps its LDAP settings in the NSSM env, not in
appsettings.json(unlike vd03). The base file'sGwAdmin/GwReadermerge with the envGroupToRole, so they were still removed from the file. The repo upgrade script carries the liveappsettings*.jsonforward, so future redeploys keep the edit. - OtOpcUa's first start after a pair stop took 37.8 s (vq04, alone in the mesh); the pair outage was ~56 s against
~37 s on 11-13. The swap itself had no
file in useretry this time (the #908 fix is in the cdf433a3 installer). - Noise seen, not caused here: OtOpcUa "Mesh PSK secret
ot-mesh-psk-devfor cluster DEV is absent" (QA has no DEV cluster mesh peer); HG plaintext/relative-path production-readiness warnings.
Rollback (per app)
Keys and files: E:\ApiInstall\_ldapcut-20261119T152947 (vq03) / …T152958 (vq04) (reg-<service>.reg, <app>_app_appsettings*.json).
- HG: the pack's
rollback.ps1(→app.bak.20261119T154443vq03 /…T154740vq04), thenreg import reg-HistorianGateway.reg, restart. - mxgw:
Upgrade-MxGatewayService.ps1 -Rollback(stamps20261119-155641vq03 /20261119-155819vq04), thenreg import reg-MxAccessGw.regand restoreMxAccessGateway_app_appsettings.json, restart. - ScadaBridge:
rollback-swapaction (app.bak.20261119T1600, Central env from_envbak-ScadaBridge-Central-20261119T1600.reg), start Central vq03, vq04, then the Sites. No DB restore needed. Keep theUG_WW_*rows. - OtOpcUa: stop both; swap
app.bak.20261119-pre-cdf433a3back;reg import reg-OtOpcUaHost.regon each; start both.
The seeded secrets are inert once nothing references them. The box-local GLAuth (ScadaBridge-LDAP) is still running
until the soak ends.
Automation identity
QA automation now uses NAM\dohertj2 (HTTP Basic / --password-file; Keychain winrm-cli/vq03). No checked-in QA
script used multi-role; the remaining multi-role users are the local Docker rig scripts, which stay on GLAuth.
Files
preflight.ps1, seed-secret.ps1, env-edit.ps1 (+ rendered env-hg.ps1, env-mxgw.ps1, env-otopcua.ps1),
mxgw-upgrade.tmpl.ps1, sb-qa.tmpl.ps1, otopcua-prepare.ps1, otopcua-swap.ps1, otopcua-swap2.ps1,
otopcua-start.ps1. Login probes reused ../2026-11-19-vd03-nam-ldap/login-test.sh, sb-cli.sh, otelgrep.py.