deploy(ldap): QA vq03/vq04 NAM AD cutover executed with the DEV builds — HG, mxgw, ScadaBridge, OtOpcUa (record + scripts)
Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
# 2026-11-19: QA vq03/vq04 NAM AD LDAP cutover + DEV builds (HistorianGateway, MxAccessGateway, ScadaBridge, OtOpcUa)
|
||||
|
||||
**Owner go (2026-11-19):** "do the qa ldap plan, winrm usage approved". Plan:
|
||||
[`docs/plans/2026-11-15-nam-ldap-cutover.md`](../../plans/2026-11-15-nam-ldap-cutover.md) §0Q (svc_igntadmin interim, DEV builds,
|
||||
one window). DEV record: [`../2026-11-19-vd03-nam-ldap/`](../2026-11-19-vd03-nam-ldap/README.md). Umbrella: scadaproj#6.
|
||||
|
||||
**Result: all four apps on both QA boxes now run the DEV builds and sign users in against NAM AD** over LDAPS (nam1, nam2
|
||||
as fallback), bound as `svc_igntadmin@nam.zimmer.com` with the password from `${secret:nam-ldap-svc-password}` in each
|
||||
app's own store. The GLAuth `multi-role` identity no longer signs in anywhere on QA. **Zero service-account bind
|
||||
failures** and zero LDAP warnings in either box's OTel log export for the whole window.
|
||||
|
||||
| App | Build | Where the LDAP config now lives |
|
||||
|---|---|---|
|
||||
| HistorianGateway (per box) | `6365f34` → **`f84af29`** | NSSM env `Ldap__*` + `Dashboard__GroupToRole__*` |
|
||||
| MxAccessGateway (per box) | `76a71bc8` → **`e74918cc`** | NSSM env `MxGateway__Ldap__*` + `MxGateway__Dashboard__GroupToRole__*`; base `app\appsettings.json` lost `GwAdmin`/`GwReader` |
|
||||
| ScadaBridge (pair) | `061432c8` → **`2e0ce81e`** | each Central's `app\appsettings.Central.json` (`ScadaBridge:Security:Ldap`); roles in `LdapGroupMappings` |
|
||||
| OtOpcUa (Warm pair) | `76e73e7d` → **`cdf433a3`** | service env `Security__Ldap__*` (+ `Security__Cookie__RequireHttpsCookie=false`) |
|
||||
|
||||
Every artifact is the one DEV runs, pulled from vd03's stage dirs over SMB and re-hashed (HG zip `1595a828…`, mxgw
|
||||
`f0999b75…`, OtOpcUa `c89ba972…`, ScadaBridge `6c706b0a…`). No app had a schema change.
|
||||
|
||||
## Steps (EST)
|
||||
|
||||
| Time | Step |
|
||||
|---|---|
|
||||
| 15:29 | **Phase 0** (`preflight.ps1`, both boxes): nam1/nam2 reachable on 636 + 389; `zimmer-rootca` in `LocalMachine\Root` (`78D941D7…`, exp 2036); LDAPS chains validate (TLS 1.3, no policy errors). Every service key and `appsettings*.json` → `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04), protected DACL. (The first run died on a parsing bug after one `reg export`; its `…T152920`/`…T152930` dirs hold only that file.) OtOpcUa Primary **vq04**, ScadaBridge active central **vq03**. HG and mxgw already had `RequireHttpsCookie=false` in env; OtOpcUa had no cookie setting |
|
||||
| 15:35 | Staged on both boxes: the four builds under `E:\psrem-stage\{hg-f84af29,mxgw-e74918cc,otopcua-cdf433a3,sb-2e0ce81e}`, the Secrets CLI 0.6.6 at `E:\ApiInstall\_tools\secret-0.6.6\`. Hashes checked on box |
|
||||
| 15:37 | **ScadaBridge role rows by CLI** as GLAuth `multi-role` against the active central (audited path): ids 6–10, `UG_WW_APP_ADMIN_QA`→Administrator, `UG_WW_APP_IDE_QA`→Designer, `UG_WW_APP_IDE_RO_QA`→Viewer, **`UG_WW_HIST_ADMIN_QA`→Deployer (interim)**, `UG_WW_BTMGR_RO_QA`→Viewer |
|
||||
| 15:41 | **Secret seeded** (`seed-secret.ps1`; each store's path and KEK read from its own service env): HG, mxgw, OtOpcUa on each box (SQLite, KEK files under `C:\ProgramData\ZB\kek\`), ScadaBridge **once**, into the shared SQL Server store (`zbsecrets` schema; now 1 row). Read-back matched in all 7; the value file was ACL'd and deleted. Both Centrals' `scadabridge.kek` are identical (`A3BD293A…`). See finding 2 for the SQL Server seeding |
|
||||
| 15:43–15:48 | **HistorianGateway** vq03 then vq04: `env-hg.ps1` (34 → 41 entries: NAM `Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then the f84af29 pack (`preflight.ps1` 0 FAIL / 0 WARN incl. a side-by-side boot with the new env, `swap.ps1`, `verify.ps1` 0 FAIL). Downtime **6.6 s / 7.7 s** |
|
||||
| 15:56–15:59 | **MxAccessGateway** vq03 then vq04: `env-mxgw.ps1` (33 → 38: NAM `MxGateway__Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then `mxgw-upgrade.tmpl.ps1` (puts the NAM base `appsettings.json` in place, then the repo upgrade script with the `START_PENDING` try/catch patch from vd03; it carries the live appsettings into the new `app\`). Dry run first on vq03. Env fingerprint unchanged through stop/swap/start/60 s stability on both |
|
||||
| 16:00 | **ScadaBridge** `prep` both nodes; DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak` (VERIFYONLY ok; 65 migrations) |
|
||||
| 16:00:53–16:04:14 | ScadaBridge window (`sb-qa.tmpl.ps1`): stop Site vq03, vq04, Central vq03, vq04; swap both (binaries + each node's own NAM `appsettings.Central.json`; `ScadaBridge__Security__Ldap__ServiceAccountPassword` removed, 15 → 14 env entries); start Central vq03 (16:03:17), vq04 (16:03:37), Site vq03 (16:03:55), vq04 (16:04:14). **Site data path ~2 min 17 s, central API ~55 s** |
|
||||
| 16:05–16:10 | **OtOpcUa** (`otopcua-prepare/swap2/start.ps1`, the 11-13 QA pack retargeted): prepare both (1410 files, deps ok, ConfigDb `OtOpcUaConfig_pre-cdf433a3_20261119.bak` + VERIFYONLY on vq03, 9 migrations, isolation `-Verify` OK); swap the Standby vq03 (`-NoStart`), `env-otopcua.ps1` (51 → 61); swap vq04 (stop 16:08:33), env (50 → 60); start vq04 (ready 16:09:29, 37.8 s), vq03 (ready 16:09:49, 8 s). Full isolation `-Verify` **OK 21/21** on both. **Pair outage ~56 s** |
|
||||
|
||||
## Mappings now on QA
|
||||
|
||||
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|
||||
|---|---|---|---|---|
|
||||
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
|
||||
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
|
||||
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
|
||||
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
|
||||
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
|
||||
|
||||
ScadaBridge still holds the five `SCADA-*` seed rows (ids 1–5); they grant nothing in NAM.
|
||||
|
||||
## Verification
|
||||
|
||||
| Check | HG `:5220` | mxgw `:5130` | ScadaBridge `:8085` | OtOpcUa `:9000` |
|
||||
|---|---|---|---|---|
|
||||
| Anonymous `GET /` → login | 302 both | 302 both | 302 | 302 both |
|
||||
| Bogus user refused | ✓ both | ✓ both | ✓ (GLAuth `multi-role` refused too, UI + CLI) | ✓ both |
|
||||
| `dohertj2` signs in | ✓ both | ✓ both | ✓ cookie; `/auth/token` 200 roles **Administrator+Designer+Deployer**; CLI (HTTP Basic) `instance list` (11) + admin `role-mapping list` | ✓ cookie, `GET /` 200 (no redirect loop); `/auth/token` 200 roles **Viewer+Designer+Administrator** |
|
||||
| After | pack `verify.ps1` 0 FAIL both; `ReadAggregate` gate as expected | `/health/ready` 200, env unchanged | central + site ready on both; active central vq03; site-a both nodes online; **MxGateway Connected 2196/2196 Good**; **WarmCanary Connected (to the Primary, vq04) 2/2 Good** after the OtOpcUa restart; LocalDb replication connected | **vq04 Primary 250 (`warm_role` 1), vq03 Standby 90**; `localdb_oplog_depth` 0 both; script workers up (shards 0, 1) both |
|
||||
|
||||
Overview `:5320` untouched. **Not done, by owner instruction:** DC-failover checks and firewall rules. **Not done:** the
|
||||
optional oswaltk Viewer-only login (step 8; needs that user).
|
||||
|
||||
## Findings
|
||||
|
||||
1. **HG pack assumed QA was on `fa9cf31`.** QA has run `6365f34` since 11-13. The pack's QA source SHA was patched
|
||||
(preflight/swap) before the run; nothing else in the pack needed changing.
|
||||
2. **The headless Secrets CLI cannot write the SQL Server store.** `secret set` registers only `AddZbSecrets` (SQLite);
|
||||
the interactive console does honour `Secrets:SqlServer`, but needs a TTY. ScadaBridge's Central uses
|
||||
`Secrets:Store:Provider=SqlServer`, so a one-off CLI build (0.6.6 + "if `Secrets:Store:Provider=SqlServer`, call
|
||||
`AddZbSecretsSqlServerStore`", mirroring ScadaBridge's own registration) seeded it, staged at
|
||||
`E:\ApiInstall\_tools\secret-0.6.6-sql\` on vq03. **Follow-up:** make the headless CLI honour the provider key the
|
||||
same way, in `ZB.MOM.WW.Secrets`.
|
||||
3. **mxgw on QA keeps its LDAP settings in the NSSM env**, not in `appsettings.json` (unlike vd03). The base file's
|
||||
`GwAdmin`/`GwReader` merge with the env `GroupToRole`, so they were still removed from the file. The repo upgrade
|
||||
script carries the live `appsettings*.json` forward, so future redeploys keep the edit.
|
||||
4. **OtOpcUa's first start after a pair stop took 37.8 s** (vq04, alone in the mesh); the pair outage was ~56 s against
|
||||
~37 s on 11-13. The swap itself had no `file in use` retry this time (the #908 fix is in the cdf433a3 installer).
|
||||
5. Noise seen, not caused here: OtOpcUa "Mesh PSK secret `ot-mesh-psk-dev` for cluster DEV is absent" (QA has no DEV
|
||||
cluster mesh peer); HG plaintext/relative-path production-readiness warnings.
|
||||
|
||||
## Rollback (per app)
|
||||
|
||||
Keys and files: `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04) (`reg-<service>.reg`, `<app>_app_appsettings*.json`).
|
||||
- **HG:** the pack's `rollback.ps1` (→ `app.bak.20261119T154443` vq03 / `…T154740` vq04), then `reg import reg-HistorianGateway.reg`, restart.
|
||||
- **mxgw:** `Upgrade-MxGatewayService.ps1 -Rollback` (stamps `20261119-155641` vq03 / `20261119-155819` vq04), then `reg import reg-MxAccessGw.reg` and restore `MxAccessGateway_app_appsettings.json`, restart.
|
||||
- **ScadaBridge:** `rollback-swap` action (`app.bak.20261119T1600`, Central env from `_envbak-ScadaBridge-Central-20261119T1600.reg`), start Central vq03, vq04, then the Sites. No DB restore needed. Keep the `UG_WW_*` rows.
|
||||
- **OtOpcUa:** stop both; swap `app.bak.20261119-pre-cdf433a3` back; `reg import reg-OtOpcUaHost.reg` on each; start both.
|
||||
|
||||
The seeded secrets are inert once nothing references them. The box-local GLAuth (`ScadaBridge-LDAP`) is still running
|
||||
until the soak ends.
|
||||
|
||||
## Automation identity
|
||||
|
||||
QA automation now uses **`NAM\dohertj2`** (HTTP Basic / `--password-file`; Keychain `winrm-cli/vq03`). No checked-in QA
|
||||
script used `multi-role`; the remaining `multi-role` users are the local Docker rig scripts, which stay on GLAuth.
|
||||
|
||||
## Files
|
||||
|
||||
`preflight.ps1`, `seed-secret.ps1`, `env-edit.ps1` (+ rendered `env-hg.ps1`, `env-mxgw.ps1`, `env-otopcua.ps1`),
|
||||
`mxgw-upgrade.tmpl.ps1`, `sb-qa.tmpl.ps1`, `otopcua-prepare.ps1`, `otopcua-swap.ps1`, `otopcua-swap2.ps1`,
|
||||
`otopcua-start.ps1`. Login probes reused `../2026-11-19-vd03-nam-ldap/login-test.sh`, `sb-cli.sh`, `otelgrep.py`.
|
||||
Reference in New Issue
Block a user