docs(ldap): NAM plan §0Q — QA vq03/vq04 cutover (svc_igntadmin interim, DEV builds, one window)

This commit is contained in:
Joseph Doherty
2026-11-19 15:25:27 -05:00
parent c80d5e79a5
commit 85915d8a57
2 changed files with 140 additions and 4 deletions
+139 -3
View File
@@ -10,7 +10,7 @@
**EXECUTED ON DEV 2026-11-19** (§0): all four apps on vd03 log in against NAM AD. Record:
[`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md). The DC-failover checks
(§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's `RequireHttpsCookie`
on a plain-HTTP AdminUI (record, finding 1). QA not started.
on a plain-HTTP AdminUI (record, finding 1). **QA plan: §0Q** (written 2026-11-19, not executed; QA hold stands).
**Updated 2026-11-19:** §0 is the **DEV vd03 execution plan using the groups and accounts that exist today**, so
it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full
prerequisites in §5–§7, and the QA deploy hold still stands.
@@ -115,8 +115,144 @@ All remote steps stay inside the `winrm` deploy scope (§7). One app at a time;
**Stop rule (§7.6 rule 6) for this run:** the account is Ignition's, so stop at the **first** `service-bind`
warning, not the second. Ask IT for the account's lockout state before any retry.
**Soak:** a few working days of real logins on DEV. QA does **not** start from this section: it still needs
scadaproj#58, D3 and D6, and the owner lifting the QA hold.
**Soak:** a few working days of real logins on DEV.
---
## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19, NOT executed)
**Owner decisions (2026-11-19):**
1. QA binds as **`svc_igntadmin`** as well (interim). This supersedes A6 and the §7 note "QA never runs on
`svc_igntadmin`". Both tiers move to the scadaproj#58 account together when it arrives: re-seed
`nam-ldap-svc-password` in every store, then a rolling restart per app.
2. **QA gets exactly the builds DEV runs.**
3. **One window:** each app's code deploy and its LDAP change share one restart.
**Gate:** the owner lifts the QA hold and names the window. **Tell the Ignition team before the window:** QA adds
8 more stores that reference their admin account.
### 0Q.1 What QA holds today
Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP).
| DEV-proven groups | QA members (2026-11-19) |
|---|---|
| `UG_WW_APP_ADMIN_QA` (6) | chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
| `UG_WW_APP_IDE_QA` (6) | the same 6 |
| `UG_WW_APP_IDE_RO_QA` (1) | oswaltk (**Viewer only**, so a live Viewer test is possible with them) |
| `UG_WW_HIST_ADMIN_QA` (6) | **the same 6 as `APP_ADMIN_QA`**, so the §0.2 Deployer interim carries over |
| `UG_WW_BTMGR_RO_QA` (0) | empty; mapped per §4 anyway (grants nobody anything) |
**Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP.** Anonymous `GET /`
returns 302 → `/login` on `:5220`, `:5130`, `:8085` and `:9000`. So QA is a **directory swap**, not a "turn login on". The
GLAuth `multi-role` automation identity stops working at the cutover; the D3 interim is `dohertj2`, as on DEV.
| App | QA build now | → target (= DEV) | Where its LDAP config lives (QA) | Notes |
|---|---|---|---|---|
| HistorianGateway (per box) | `6365f34` | `f84af29` (#136) | **NSSM env** `Ldap__*`, `Dashboard__GroupToRole__SCADA-*` [I] | `app\appsettings.json` has `Dashboard:RequireHttpsCookie=true`, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below) |
| MxAccessGateway (per box) | `76a71bc8` | `e74918cc` | `app\appsettings.json`: GLAuth `localhost:3893`, `dc=zb,dc=local`, `${secret:ldap/mxgateway/bind}`, **base `GwAdmin`/`GwReader`** | Phase 0 records the environment name and any overlay. Delete `GwAdmin`/`GwReader` from the base file (§4.2) |
| ScadaBridge (pair) | `061432c8` | `2e0ce81e` (#144/#145) | `app\appsettings.Central.json` (GLAuth `127.0.0.1:3893`, `dc=scadabridge`) + Central env password | **`Secrets:Store:Provider=SqlServer`**: one shared store, so seed once. `RequireHttpsCookie=false` already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other |
| OtOpcUa (Warm pair) | `76e73e7d` | `cdf433a3` (#913; 83 commits, **no migration**) | service env [I] | Warm redundancy (`Mode=Warm`); Phase 0 records which node is Primary. Needs the cookie decision (finding 1) |
Secrets master keys: the shipped files say `Source=Environment` (`ZB_SECRETS_MASTER_KEY`) for HG, mxgw and OtOpcUa. The
service env may override that. The seed script reads each service's own env, so it never assumes.
### 0Q.2 Findings carried from DEV
1. **Plain-HTTP cookies.** OtOpcUa's auth cookie is `Secure` by default, so its AdminUI login loops over HTTP. On DEV
the fix was `Security__Cookie__RequireHttpsCookie=false`. QA OtOpcUa login works today over HTTP, so Phase 0 checks
whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HG `Dashboard:RequireHttpsCookie`
the same way.
2. **Secrets CLI.** The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store
schema or envelope change), copied to `E:\ApiInstall\_tools\secret-0.6.6\` on each box.
3. **Do not** add firewall rules or run DC-failover drills (owner, 2026-11-19).
4. Logs: the apps' own files may be empty. Verification reads the box OTel collector export
(`E:\ApiInstall\OtelCollector\data\logs.jsonl`) over SMB.
### 0Q.3 Mappings (QA suffix)
As §4, plus the Deployer interim:
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
Remove every GLAuth mapping: HG `SCADA-*` env entries and mxgw `GwAdmin`/`GwReader`. The ScadaBridge `SCADA-*` seed rows
stay; they grant nothing in NAM.
### 0Q.4 The window
All remote steps are `winrm` deploy scope on vq03/vq04, one app at a time, in the canary order. **Stop at the first
`service-bind` warning** (§0.4 stop rule). Each app's code deploy and LDAP change share one restart.
0. **Before the window (from the Mac, no remote exec):**
- stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB;
- **add the ScadaBridge role rows with the CLI** while QA still logs in through GLAuth. Log in as `multi-role`,
with `--url` pointing at the **active** central. This path is audited and invalidates the cache, unlike the SQL path DEV
had to use. Add the five rows from §0Q.3; they grant nothing until the cutover.
1. **Phase 0 per box** (`preflight.ps1` from the DEV record, retargeted to the QA layout `E:\ApiInstall\<App>\app`
and the service names):
- DC reach on 636 and `zimmer-rootca` trust;
- back up every service key and every `appsettings*.json` to `E:\ApiInstall\_ldapcut-<ts>\`;
- record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa `Secrets:Replication:Enabled`, the
OtOpcUa Primary/Standby and ServiceLevels, and the `LdapGroupRoleMapping` overlay rows;
- read each app's secrets provider and master-key source from its service env.
2. **Seed `nam-ldap-svc-password`** (`seed-secret.ps1`, generalised to read each service's own `Secrets__*` and
master-key env):
- HG and mxgw: per box;
- OtOpcUa: per box, or once if replication is on;
- ScadaBridge: **once**, into the shared SQL store, using the active Central's provider settings.
Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted.
3. **HistorianGateway**, vq03 then vq04 (independent single nodes; this is the canary):
- deploy `f84af29` with HG's own procedure (`HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/` retargeted to
QA);
- before the start, rewrite the NSSM env: `Ldap__*` → §6, `Dashboard__GroupToRole__*` → §0Q.3, and the cookie per
finding 1.
Verify on each box: bogus refused, `dohertj2` → Administrator, no `system-side` lines. Then the next box.
4. **MxAccessGateway**, vq03 then vq04:
- deploy `e74918cc` with mxgw's procedure. It will replace `app\appsettings.json`, so apply the edit to the **new**
file before the start: §6 block, QA `GroupToRole`, delete `GwAdmin`/`GwReader`, no GLAuth pin;
- mind the nssm `START_PENDING` abort in the repo upgrade script (memory note: use the try/catch fix as on vd03).
Verify per box as in step 3.
5. **ScadaBridge pair**: deploy `2e0ce81e` with the QA pair procedure (`deploy-prep/2026-11-15-143/qa.tmpl.ps1` actions
minus `migrate`):
- stop both Sites, then both Centrals, so the pair goes down together;
- swap the binaries and edit **each** Central's `appsettings.Central.json`: §6 block, drop any `Auth` block;
- remove `ScadaBridge__Security__Ldap__ServiceAccountPassword` from **both** Central envs;
- start the Centrals (vq03 first), then the Sites.
Verify:
- `dohertj2` gets Administrator+Designer+Deployer at `/auth/login`, at `/auth/token` and on the CLI against the
active central;
- bogus refused;
- LocalDb sync connected; Z28062–Z28069 at their Good baseline; `WarmCanary` connected; one active per pair.
6. **OtOpcUa Warm pair**: deploy `cdf433a3` with OtOpcUa's QA procedure (the 2026-11-13 `76e73e7d` pack: stop-both /
start-both for a binary upgrade, worker isolation `-Reapply`/`-Verify`).
- **Before the start**, add to **both** nodes' env: the 18 `Security__Ldap__*` entries from the DEV record (QA suffix
in `GroupToRole`), `Security__Auth__DisableLogin` left `false`, and the cookie per finding 1.
- Verify:
- ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads `BadOutOfService`;
- LocalDb sync connected; the script-runner canary Good;
- `dohertj2` → Viewer+Designer+Administrator via cookie and `/auth/token`; bogus refused;
- the ScadaBridge `WarmCanary` reconnects.
7. **Re-point QA automation** from `multi-role` to `dohertj2`: the qa pack scripts, drills, `apply.sh`.
8. **Optional:** oswaltk's Viewer-only login (the only Viewer-only member).
9. **Record** in `docs/deploy/2026-11-<dd>-qa-nam-ldap/`, plus a deploy record in each repo.
**Outages:** HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on
the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is
also a binary upgrade, and the pairs can't roll those.
**Rollback:** per app, restore the `_ldapcut-<ts>` keys and files. Binaries roll back with each repo's own procedure
(no app here has a schema change). Keep the ScadaBridge `UG_WW_*` rows. The box-local GLAuth stays running until the
soak ends.
**Goal:** move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth
test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair.