From 85915d8a576b19ad16e5f2f4ffad4c4d37455403 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 19 Nov 2026 15:25:27 -0500 Subject: [PATCH] =?UTF-8?q?docs(ldap):=20NAM=20plan=20=C2=A70Q=20=E2=80=94?= =?UTF-8?q?=20QA=20vq03/vq04=20cutover=20(svc=5Figntadmin=20interim,=20DEV?= =?UTF-8?q?=20builds,=20one=20window)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/plans/2026-11-15-nam-ldap-cutover.md | 142 +++++++++++++++++++++- ldap_mapping.md | 2 +- 2 files changed, 140 insertions(+), 4 deletions(-) diff --git a/docs/plans/2026-11-15-nam-ldap-cutover.md b/docs/plans/2026-11-15-nam-ldap-cutover.md index a160200..2d16c35 100644 --- a/docs/plans/2026-11-15-nam-ldap-cutover.md +++ b/docs/plans/2026-11-15-nam-ldap-cutover.md @@ -10,7 +10,7 @@ **EXECUTED ON DEV 2026-11-19** (§0): all four apps on vd03 log in against NAM AD. Record: [`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md). The DC-failover checks (§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's `RequireHttpsCookie` -on a plain-HTTP AdminUI (record, finding 1). QA not started. +on a plain-HTTP AdminUI (record, finding 1). **QA plan: §0Q** (written 2026-11-19, not executed; QA hold stands). **Updated 2026-11-19:** §0 is the **DEV vd03 execution plan using the groups and accounts that exist today**, so it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full prerequisites in §5–§7, and the QA deploy hold still stands. @@ -115,8 +115,144 @@ All remote steps stay inside the `winrm` deploy scope (§7). One app at a time; **Stop rule (§7.6 rule 6) for this run:** the account is Ignition's, so stop at the **first** `service-bind` warning, not the second. Ask IT for the account's lockout state before any retry. -**Soak:** a few working days of real logins on DEV. QA does **not** start from this section: it still needs -scadaproj#58, D3 and D6, and the owner lifting the QA hold. +**Soak:** a few working days of real logins on DEV. + +--- + +## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19, NOT executed) + +**Owner decisions (2026-11-19):** +1. QA binds as **`svc_igntadmin`** as well (interim). This supersedes A6 and the §7 note "QA never runs on + `svc_igntadmin`". Both tiers move to the scadaproj#58 account together when it arrives: re-seed + `nam-ldap-svc-password` in every store, then a rolling restart per app. +2. **QA gets exactly the builds DEV runs.** +3. **One window:** each app's code deploy and its LDAP change share one restart. + +**Gate:** the owner lifts the QA hold and names the window. **Tell the Ignition team before the window:** QA adds +8 more stores that reference their admin account. + +### 0Q.1 What QA holds today + +Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP). + +| DEV-proven groups | QA members (2026-11-19) | +|---|---| +| `UG_WW_APP_ADMIN_QA` (6) | chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas | +| `UG_WW_APP_IDE_QA` (6) | the same 6 | +| `UG_WW_APP_IDE_RO_QA` (1) | oswaltk (**Viewer only**, so a live Viewer test is possible with them) | +| `UG_WW_HIST_ADMIN_QA` (6) | **the same 6 as `APP_ADMIN_QA`**, so the §0.2 Deployer interim carries over | +| `UG_WW_BTMGR_RO_QA` (0) | empty; mapped per §4 anyway (grants nobody anything) | + +**Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP.** Anonymous `GET /` +returns 302 → `/login` on `:5220`, `:5130`, `:8085` and `:9000`. So QA is a **directory swap**, not a "turn login on". The +GLAuth `multi-role` automation identity stops working at the cutover; the D3 interim is `dohertj2`, as on DEV. + +| App | QA build now | → target (= DEV) | Where its LDAP config lives (QA) | Notes | +|---|---|---|---|---| +| HistorianGateway (per box) | `6365f34` | `f84af29` (#136) | **NSSM env** `Ldap__*`, `Dashboard__GroupToRole__SCADA-*` [I] | `app\appsettings.json` has `Dashboard:RequireHttpsCookie=true`, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below) | +| MxAccessGateway (per box) | `76a71bc8` | `e74918cc` | `app\appsettings.json`: GLAuth `localhost:3893`, `dc=zb,dc=local`, `${secret:ldap/mxgateway/bind}`, **base `GwAdmin`/`GwReader`** | Phase 0 records the environment name and any overlay. Delete `GwAdmin`/`GwReader` from the base file (§4.2) | +| ScadaBridge (pair) | `061432c8` | `2e0ce81e` (#144/#145) | `app\appsettings.Central.json` (GLAuth `127.0.0.1:3893`, `dc=scadabridge`) + Central env password | **`Secrets:Store:Provider=SqlServer`**: one shared store, so seed once. `RequireHttpsCookie=false` already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other | +| OtOpcUa (Warm pair) | `76e73e7d` | `cdf433a3` (#913; 83 commits, **no migration**) | service env [I] | Warm redundancy (`Mode=Warm`); Phase 0 records which node is Primary. Needs the cookie decision (finding 1) | + +Secrets master keys: the shipped files say `Source=Environment` (`ZB_SECRETS_MASTER_KEY`) for HG, mxgw and OtOpcUa. The +service env may override that. The seed script reads each service's own env, so it never assumes. + +### 0Q.2 Findings carried from DEV + +1. **Plain-HTTP cookies.** OtOpcUa's auth cookie is `Secure` by default, so its AdminUI login loops over HTTP. On DEV + the fix was `Security__Cookie__RequireHttpsCookie=false`. QA OtOpcUa login works today over HTTP, so Phase 0 checks + whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HG `Dashboard:RequireHttpsCookie` + the same way. +2. **Secrets CLI.** The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store + schema or envelope change), copied to `E:\ApiInstall\_tools\secret-0.6.6\` on each box. +3. **Do not** add firewall rules or run DC-failover drills (owner, 2026-11-19). +4. Logs: the apps' own files may be empty. Verification reads the box OTel collector export + (`E:\ApiInstall\OtelCollector\data\logs.jsonl`) over SMB. + +### 0Q.3 Mappings (QA suffix) + +As §4, plus the Deployer interim: + +| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway | +|---|---|---|---|---| +| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator | +| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — | +| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer | +| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator | +| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer | + +Remove every GLAuth mapping: HG `SCADA-*` env entries and mxgw `GwAdmin`/`GwReader`. The ScadaBridge `SCADA-*` seed rows +stay; they grant nothing in NAM. + +### 0Q.4 The window + +All remote steps are `winrm` deploy scope on vq03/vq04, one app at a time, in the canary order. **Stop at the first +`service-bind` warning** (§0.4 stop rule). Each app's code deploy and LDAP change share one restart. + +0. **Before the window (from the Mac, no remote exec):** + - stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB; + - **add the ScadaBridge role rows with the CLI** while QA still logs in through GLAuth. Log in as `multi-role`, + with `--url` pointing at the **active** central. This path is audited and invalidates the cache, unlike the SQL path DEV + had to use. Add the five rows from §0Q.3; they grant nothing until the cutover. +1. **Phase 0 per box** (`preflight.ps1` from the DEV record, retargeted to the QA layout `E:\ApiInstall\\app` + and the service names): + - DC reach on 636 and `zimmer-rootca` trust; + - back up every service key and every `appsettings*.json` to `E:\ApiInstall\_ldapcut-\`; + - record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa `Secrets:Replication:Enabled`, the + OtOpcUa Primary/Standby and ServiceLevels, and the `LdapGroupRoleMapping` overlay rows; + - read each app's secrets provider and master-key source from its service env. +2. **Seed `nam-ldap-svc-password`** (`seed-secret.ps1`, generalised to read each service's own `Secrets__*` and + master-key env): + - HG and mxgw: per box; + - OtOpcUa: per box, or once if replication is on; + - ScadaBridge: **once**, into the shared SQL store, using the active Central's provider settings. + + Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted. +3. **HistorianGateway**, vq03 then vq04 (independent single nodes; this is the canary): + - deploy `f84af29` with HG's own procedure (`HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/` retargeted to + QA); + - before the start, rewrite the NSSM env: `Ldap__*` → §6, `Dashboard__GroupToRole__*` → §0Q.3, and the cookie per + finding 1. + + Verify on each box: bogus refused, `dohertj2` → Administrator, no `system-side` lines. Then the next box. +4. **MxAccessGateway**, vq03 then vq04: + - deploy `e74918cc` with mxgw's procedure. It will replace `app\appsettings.json`, so apply the edit to the **new** + file before the start: §6 block, QA `GroupToRole`, delete `GwAdmin`/`GwReader`, no GLAuth pin; + - mind the nssm `START_PENDING` abort in the repo upgrade script (memory note: use the try/catch fix as on vd03). + + Verify per box as in step 3. +5. **ScadaBridge pair**: deploy `2e0ce81e` with the QA pair procedure (`deploy-prep/2026-11-15-143/qa.tmpl.ps1` actions + minus `migrate`): + - stop both Sites, then both Centrals, so the pair goes down together; + - swap the binaries and edit **each** Central's `appsettings.Central.json`: §6 block, drop any `Auth` block; + - remove `ScadaBridge__Security__Ldap__ServiceAccountPassword` from **both** Central envs; + - start the Centrals (vq03 first), then the Sites. + + Verify: + - `dohertj2` gets Administrator+Designer+Deployer at `/auth/login`, at `/auth/token` and on the CLI against the + active central; + - bogus refused; + - LocalDb sync connected; Z28062–Z28069 at their Good baseline; `WarmCanary` connected; one active per pair. +6. **OtOpcUa Warm pair**: deploy `cdf433a3` with OtOpcUa's QA procedure (the 2026-11-13 `76e73e7d` pack: stop-both / + start-both for a binary upgrade, worker isolation `-Reapply`/`-Verify`). + - **Before the start**, add to **both** nodes' env: the 18 `Security__Ldap__*` entries from the DEV record (QA suffix + in `GroupToRole`), `Security__Auth__DisableLogin` left `false`, and the cookie per finding 1. + - Verify: + - ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads `BadOutOfService`; + - LocalDb sync connected; the script-runner canary Good; + - `dohertj2` → Viewer+Designer+Administrator via cookie and `/auth/token`; bogus refused; + - the ScadaBridge `WarmCanary` reconnects. +7. **Re-point QA automation** from `multi-role` to `dohertj2`: the qa pack scripts, drills, `apply.sh`. +8. **Optional:** oswaltk's Viewer-only login (the only Viewer-only member). +9. **Record** in `docs/deploy/2026-11-
-qa-nam-ldap/`, plus a deploy record in each repo. + +**Outages:** HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on +the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is +also a binary upgrade, and the pairs can't roll those. + +**Rollback:** per app, restore the `_ldapcut-` keys and files. Binaries roll back with each repo's own procedure +(no app here has a schema change). Keep the ScadaBridge `UG_WW_*` rows. The box-local GLAuth stays running until the +soak ends. **Goal:** move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair. diff --git a/ldap_mapping.md b/ldap_mapping.md index 7cf7649..4b23c99 100644 --- a/ldap_mapping.md +++ b/ldap_mapping.md @@ -81,7 +81,7 @@ made. One role per cell. |---|---|---|---|---|---| | `UG_WW_APP_ADMIN_` | Administrator | Administrator | Administrator | Administrator | | | `UG_WW_APP_IDE_` | Designer | Designer | Viewer | — | **Was "Designer + Deployer" in ScadaBridge.** Resolved: Designer here; `Deployer` comes from the new group below. mxgw has no Designer, so Viewer. | -| `UG_WW_APP_DEPLOY_` | — | Deployer | — | — | **NEW group: owner/IT decision, request by ServiceNow ticket** (cutover plan D6). Until it exists, no NAM user can deploy in ScadaBridge, so the ScadaBridge cutover waits for it. **DEV interim (2026-11-19, plan §0.2):** ScadaBridge DEV maps `UG_WW_HIST_ADMIN_DEV` → `Deployer` (same 7 members as `APP_ADMIN_DEV`) until this group exists. | +| `UG_WW_APP_DEPLOY_` | — | Deployer | — | — | **NEW group: owner/IT decision, request by ServiceNow ticket** (cutover plan D6). Until it exists, no NAM user can deploy in ScadaBridge, so the ScadaBridge cutover waits for it. **DEV interim (2026-11-19, plan §0.2):** ScadaBridge DEV maps `UG_WW_HIST_ADMIN_DEV` → `Deployer` (same 7 members as `APP_ADMIN_DEV`) until this group exists. QA plans the same with `UG_WW_HIST_ADMIN_QA` (same 6 as `APP_ADMIN_QA`; plan §0Q). | | `UG_WW_APP_IDE_RO_` | Viewer | Viewer | Viewer | Viewer | | | `UG_WW_HIST_ADMIN_` | Viewer | — | — | Administrator | OtOpcUa Viewer, because historization targets are authored in the AdminUI and visible to a historian admin, but not editable. | | `UG_WW_BTMGR_RO_` (QA/PROD only) | — | Viewer | — | Viewer | `BTMGR_RO_QA` is **empty** (08-12). |