Files
scadaproj/docs/deploy/2026-11-19-qa-nam-ldap/README.md
T

99 lines
9.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# 2026-11-19: QA vq03/vq04 NAM AD LDAP cutover + DEV builds (HistorianGateway, MxAccessGateway, ScadaBridge, OtOpcUa)
**Owner go (2026-11-19):** "do the qa ldap plan, winrm usage approved". Plan:
[`docs/plans/2026-11-15-nam-ldap-cutover.md`](../../plans/2026-11-15-nam-ldap-cutover.md) §0Q (svc_igntadmin interim, DEV builds,
one window). DEV record: [`../2026-11-19-vd03-nam-ldap/`](../2026-11-19-vd03-nam-ldap/README.md). Umbrella: scadaproj#6.
**Result: all four apps on both QA boxes now run the DEV builds and sign users in against NAM AD** over LDAPS (nam1, nam2
as fallback), bound as `svc_igntadmin@nam.zimmer.com` with the password from `${secret:nam-ldap-svc-password}` in each
app's own store. The GLAuth `multi-role` identity no longer signs in anywhere on QA. **Zero service-account bind
failures** and zero LDAP warnings in either box's OTel log export for the whole window.
| App | Build | Where the LDAP config now lives |
|---|---|---|
| HistorianGateway (per box) | `6365f34` → **`f84af29`** | NSSM env `Ldap__*` + `Dashboard__GroupToRole__*` |
| MxAccessGateway (per box) | `76a71bc8` → **`e74918cc`** | NSSM env `MxGateway__Ldap__*` + `MxGateway__Dashboard__GroupToRole__*`; base `app\appsettings.json` lost `GwAdmin`/`GwReader` |
| ScadaBridge (pair) | `061432c8` → **`2e0ce81e`** | each Central's `app\appsettings.Central.json` (`ScadaBridge:Security:Ldap`); roles in `LdapGroupMappings` |
| OtOpcUa (Warm pair) | `76e73e7d` → **`cdf433a3`** | service env `Security__Ldap__*` (+ `Security__Cookie__RequireHttpsCookie=false`) |
Every artifact is the one DEV runs, pulled from vd03's stage dirs over SMB and re-hashed (HG zip `1595a828…`, mxgw
`f0999b75…`, OtOpcUa `c89ba972…`, ScadaBridge `6c706b0a…`). No app had a schema change.
## Steps (EST)
| Time | Step |
|---|---|
| 15:29 | **Phase 0** (`preflight.ps1`, both boxes): nam1/nam2 reachable on 636 + 389; `zimmer-rootca` in `LocalMachine\Root` (`78D941D7…`, exp 2036); LDAPS chains validate (TLS 1.3, no policy errors). Every service key and `appsettings*.json` → `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04), protected DACL. (The first run died on a parsing bug after one `reg export`; its `…T152920`/`…T152930` dirs hold only that file.) OtOpcUa Primary **vq04**, ScadaBridge active central **vq03**. HG and mxgw already had `RequireHttpsCookie=false` in env; OtOpcUa had no cookie setting |
| 15:35 | Staged on both boxes: the four builds under `E:\psrem-stage\{hg-f84af29,mxgw-e74918cc,otopcua-cdf433a3,sb-2e0ce81e}`, the Secrets CLI 0.6.6 at `E:\ApiInstall\_tools\secret-0.6.6\`. Hashes checked on box |
| 15:37 | **ScadaBridge role rows by CLI** as GLAuth `multi-role` against the active central (audited path): ids 6–10, `UG_WW_APP_ADMIN_QA`→Administrator, `UG_WW_APP_IDE_QA`→Designer, `UG_WW_APP_IDE_RO_QA`→Viewer, **`UG_WW_HIST_ADMIN_QA`→Deployer (interim)**, `UG_WW_BTMGR_RO_QA`→Viewer |
| 15:41 | **Secret seeded** (`seed-secret.ps1`; each store's path and KEK read from its own service env): HG, mxgw, OtOpcUa on each box (SQLite, KEK files under `C:\ProgramData\ZB\kek\`), ScadaBridge **once**, into the shared SQL Server store (`zbsecrets` schema; now 1 row). Read-back matched in all 7; the value file was ACL'd and deleted. Both Centrals' `scadabridge.kek` are identical (`A3BD293A…`). See finding 2 for the SQL Server seeding |
| 15:43–15:48 | **HistorianGateway** vq03 then vq04: `env-hg.ps1` (34 → 41 entries: NAM `Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then the f84af29 pack (`preflight.ps1` 0 FAIL / 0 WARN incl. a side-by-side boot with the new env, `swap.ps1`, `verify.ps1` 0 FAIL). Downtime **6.6 s / 7.7 s** |
| 15:56–15:59 | **MxAccessGateway** vq03 then vq04: `env-mxgw.ps1` (33 → 38: NAM `MxGateway__Ldap__*`, QA `GroupToRole`; `SCADA-*` dropped), then `mxgw-upgrade.tmpl.ps1` (puts the NAM base `appsettings.json` in place, then the repo upgrade script with the `START_PENDING` try/catch patch from vd03; it carries the live appsettings into the new `app\`). Dry run first on vq03. Env fingerprint unchanged through stop/swap/start/60 s stability on both |
| 16:00 | **ScadaBridge** `prep` both nodes; DB backup `F:\MSSQL\BACKUP\ScadaBridge_pre-2e0ce81e_20261119T1600.bak` (VERIFYONLY ok; 65 migrations) |
| 16:00:53–16:04:14 | ScadaBridge window (`sb-qa.tmpl.ps1`): stop Site vq03, vq04, Central vq03, vq04; swap both (binaries + each node's own NAM `appsettings.Central.json`; `ScadaBridge__Security__Ldap__ServiceAccountPassword` removed, 15 → 14 env entries); start Central vq03 (16:03:17), vq04 (16:03:37), Site vq03 (16:03:55), vq04 (16:04:14). **Site data path ~2 min 17 s, central API ~55 s** |
| 16:05–16:10 | **OtOpcUa** (`otopcua-prepare/swap2/start.ps1`, the 11-13 QA pack retargeted): prepare both (1410 files, deps ok, ConfigDb `OtOpcUaConfig_pre-cdf433a3_20261119.bak` + VERIFYONLY on vq03, 9 migrations, isolation `-Verify` OK); swap the Standby vq03 (`-NoStart`), `env-otopcua.ps1` (51 → 61); swap vq04 (stop 16:08:33), env (50 → 60); start vq04 (ready 16:09:29, 37.8 s), vq03 (ready 16:09:49, 8 s). Full isolation `-Verify` **OK 21/21** on both. **Pair outage ~56 s** |
## Mappings now on QA
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
ScadaBridge still holds the five `SCADA-*` seed rows (ids 1–5); they grant nothing in NAM.
## Verification
| Check | HG `:5220` | mxgw `:5130` | ScadaBridge `:8085` | OtOpcUa `:9000` |
|---|---|---|---|---|
| Anonymous `GET /` → login | 302 both | 302 both | 302 | 302 both |
| Bogus user refused | ✓ both | ✓ both | ✓ (GLAuth `multi-role` refused too, UI + CLI) | ✓ both |
| `dohertj2` signs in | ✓ both | ✓ both | ✓ cookie; `/auth/token` 200 roles **Administrator+Designer+Deployer**; CLI (HTTP Basic) `instance list` (11) + admin `role-mapping list` | ✓ cookie, `GET /` 200 (no redirect loop); `/auth/token` 200 roles **Viewer+Designer+Administrator** |
| After | pack `verify.ps1` 0 FAIL both; `ReadAggregate` gate as expected | `/health/ready` 200, env unchanged | central + site ready on both; active central vq03; site-a both nodes online; **MxGateway Connected 2196/2196 Good**; **WarmCanary Connected (to the Primary, vq04) 2/2 Good** after the OtOpcUa restart; LocalDb replication connected | **vq04 Primary 250 (`warm_role` 1), vq03 Standby 90**; `localdb_oplog_depth` 0 both; script workers up (shards 0, 1) both |
Overview `:5320` untouched. **Not done, by owner instruction:** DC-failover checks and firewall rules. **Not done:** the
optional oswaltk Viewer-only login (step 8; needs that user).
## Findings
1. **HG pack assumed QA was on `fa9cf31`.** QA has run `6365f34` since 11-13. The pack's QA source SHA was patched
(preflight/swap) before the run; nothing else in the pack needed changing.
2. **The headless Secrets CLI cannot write the SQL Server store.** `secret set` registers only `AddZbSecrets` (SQLite);
the interactive console does honour `Secrets:SqlServer`, but needs a TTY. ScadaBridge's Central uses
`Secrets:Store:Provider=SqlServer`, so a one-off CLI build (0.6.6 + "if `Secrets:Store:Provider=SqlServer`, call
`AddZbSecretsSqlServerStore`", mirroring ScadaBridge's own registration) seeded it, staged at
`E:\ApiInstall\_tools\secret-0.6.6-sql\` on vq03. **Follow-up:** make the headless CLI honour the provider key the
same way, in `ZB.MOM.WW.Secrets`.
3. **mxgw on QA keeps its LDAP settings in the NSSM env**, not in `appsettings.json` (unlike vd03). The base file's
`GwAdmin`/`GwReader` merge with the env `GroupToRole`, so they were still removed from the file. The repo upgrade
script carries the live `appsettings*.json` forward, so future redeploys keep the edit.
4. **OtOpcUa's first start after a pair stop took 37.8 s** (vq04, alone in the mesh); the pair outage was ~56 s against
~37 s on 11-13. The swap itself had no `file in use` retry this time (the #908 fix is in the cdf433a3 installer).
5. Noise seen, not caused here: OtOpcUa "Mesh PSK secret `ot-mesh-psk-dev` for cluster DEV is absent" (QA has no DEV
cluster mesh peer); HG plaintext/relative-path production-readiness warnings.
## Rollback (per app)
Keys and files: `E:\ApiInstall\_ldapcut-20261119T152947` (vq03) / `…T152958` (vq04) (`reg-<service>.reg`, `<app>_app_appsettings*.json`).
- **HG:** the pack's `rollback.ps1` (→ `app.bak.20261119T154443` vq03 / `…T154740` vq04), then `reg import reg-HistorianGateway.reg`, restart.
- **mxgw:** `Upgrade-MxGatewayService.ps1 -Rollback` (stamps `20261119-155641` vq03 / `20261119-155819` vq04), then `reg import reg-MxAccessGw.reg` and restore `MxAccessGateway_app_appsettings.json`, restart.
- **ScadaBridge:** `rollback-swap` action (`app.bak.20261119T1600`, Central env from `_envbak-ScadaBridge-Central-20261119T1600.reg`), start Central vq03, vq04, then the Sites. No DB restore needed. Keep the `UG_WW_*` rows.
- **OtOpcUa:** stop both; swap `app.bak.20261119-pre-cdf433a3` back; `reg import reg-OtOpcUaHost.reg` on each; start both.
The seeded secrets are inert once nothing references them. The box-local GLAuth (`ScadaBridge-LDAP`) is still running
until the soak ends.
## Automation identity
QA automation now uses **`NAM\dohertj2`** (HTTP Basic / `--password-file`; Keychain `winrm-cli/vq03`). No checked-in QA
script used `multi-role`; the remaining `multi-role` users are the local Docker rig scripts, which stay on GLAuth.
## Files
`preflight.ps1`, `seed-secret.ps1`, `env-edit.ps1` (+ rendered `env-hg.ps1`, `env-mxgw.ps1`, `env-otopcua.ps1`),
`mxgw-upgrade.tmpl.ps1`, `sb-qa.tmpl.ps1`, `otopcua-prepare.ps1`, `otopcua-swap.ps1`, `otopcua-swap2.ps1`,
`otopcua-start.ps1`. Login probes reused `../2026-11-19-vd03-nam-ldap/login-test.sh`, `sb-cli.sh`, `otelgrep.py`.