docs(ldap): NAM plan §0Q — QA vq03/vq04 cutover (svc_igntadmin interim, DEV builds, one window)
This commit is contained in:
@@ -10,7 +10,7 @@
|
||||
**EXECUTED ON DEV 2026-11-19** (§0): all four apps on vd03 log in against NAM AD. Record:
|
||||
[`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md). The DC-failover checks
|
||||
(§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's `RequireHttpsCookie`
|
||||
on a plain-HTTP AdminUI (record, finding 1). QA not started.
|
||||
on a plain-HTTP AdminUI (record, finding 1). **QA plan: §0Q** (written 2026-11-19, not executed; QA hold stands).
|
||||
**Updated 2026-11-19:** §0 is the **DEV vd03 execution plan using the groups and accounts that exist today**, so
|
||||
it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full
|
||||
prerequisites in §5–§7, and the QA deploy hold still stands.
|
||||
@@ -115,8 +115,144 @@ All remote steps stay inside the `winrm` deploy scope (§7). One app at a time;
|
||||
**Stop rule (§7.6 rule 6) for this run:** the account is Ignition's, so stop at the **first** `service-bind`
|
||||
warning, not the second. Ask IT for the account's lockout state before any retry.
|
||||
|
||||
**Soak:** a few working days of real logins on DEV. QA does **not** start from this section: it still needs
|
||||
scadaproj#58, D3 and D6, and the owner lifting the QA hold.
|
||||
**Soak:** a few working days of real logins on DEV.
|
||||
|
||||
---
|
||||
|
||||
## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19, NOT executed)
|
||||
|
||||
**Owner decisions (2026-11-19):**
|
||||
1. QA binds as **`svc_igntadmin`** as well (interim). This supersedes A6 and the §7 note "QA never runs on
|
||||
`svc_igntadmin`". Both tiers move to the scadaproj#58 account together when it arrives: re-seed
|
||||
`nam-ldap-svc-password` in every store, then a rolling restart per app.
|
||||
2. **QA gets exactly the builds DEV runs.**
|
||||
3. **One window:** each app's code deploy and its LDAP change share one restart.
|
||||
|
||||
**Gate:** the owner lifts the QA hold and names the window. **Tell the Ignition team before the window:** QA adds
|
||||
8 more stores that reference their admin account.
|
||||
|
||||
### 0Q.1 What QA holds today
|
||||
|
||||
Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP).
|
||||
|
||||
| DEV-proven groups | QA members (2026-11-19) |
|
||||
|---|---|
|
||||
| `UG_WW_APP_ADMIN_QA` (6) | chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
|
||||
| `UG_WW_APP_IDE_QA` (6) | the same 6 |
|
||||
| `UG_WW_APP_IDE_RO_QA` (1) | oswaltk (**Viewer only**, so a live Viewer test is possible with them) |
|
||||
| `UG_WW_HIST_ADMIN_QA` (6) | **the same 6 as `APP_ADMIN_QA`**, so the §0.2 Deployer interim carries over |
|
||||
| `UG_WW_BTMGR_RO_QA` (0) | empty; mapped per §4 anyway (grants nobody anything) |
|
||||
|
||||
**Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP.** Anonymous `GET /`
|
||||
returns 302 → `/login` on `:5220`, `:5130`, `:8085` and `:9000`. So QA is a **directory swap**, not a "turn login on". The
|
||||
GLAuth `multi-role` automation identity stops working at the cutover; the D3 interim is `dohertj2`, as on DEV.
|
||||
|
||||
| App | QA build now | → target (= DEV) | Where its LDAP config lives (QA) | Notes |
|
||||
|---|---|---|---|---|
|
||||
| HistorianGateway (per box) | `6365f34` | `f84af29` (#136) | **NSSM env** `Ldap__*`, `Dashboard__GroupToRole__SCADA-*` [I] | `app\appsettings.json` has `Dashboard:RequireHttpsCookie=true`, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below) |
|
||||
| MxAccessGateway (per box) | `76a71bc8` | `e74918cc` | `app\appsettings.json`: GLAuth `localhost:3893`, `dc=zb,dc=local`, `${secret:ldap/mxgateway/bind}`, **base `GwAdmin`/`GwReader`** | Phase 0 records the environment name and any overlay. Delete `GwAdmin`/`GwReader` from the base file (§4.2) |
|
||||
| ScadaBridge (pair) | `061432c8` | `2e0ce81e` (#144/#145) | `app\appsettings.Central.json` (GLAuth `127.0.0.1:3893`, `dc=scadabridge`) + Central env password | **`Secrets:Store:Provider=SqlServer`**: one shared store, so seed once. `RequireHttpsCookie=false` already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other |
|
||||
| OtOpcUa (Warm pair) | `76e73e7d` | `cdf433a3` (#913; 83 commits, **no migration**) | service env [I] | Warm redundancy (`Mode=Warm`); Phase 0 records which node is Primary. Needs the cookie decision (finding 1) |
|
||||
|
||||
Secrets master keys: the shipped files say `Source=Environment` (`ZB_SECRETS_MASTER_KEY`) for HG, mxgw and OtOpcUa. The
|
||||
service env may override that. The seed script reads each service's own env, so it never assumes.
|
||||
|
||||
### 0Q.2 Findings carried from DEV
|
||||
|
||||
1. **Plain-HTTP cookies.** OtOpcUa's auth cookie is `Secure` by default, so its AdminUI login loops over HTTP. On DEV
|
||||
the fix was `Security__Cookie__RequireHttpsCookie=false`. QA OtOpcUa login works today over HTTP, so Phase 0 checks
|
||||
whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HG `Dashboard:RequireHttpsCookie`
|
||||
the same way.
|
||||
2. **Secrets CLI.** The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store
|
||||
schema or envelope change), copied to `E:\ApiInstall\_tools\secret-0.6.6\` on each box.
|
||||
3. **Do not** add firewall rules or run DC-failover drills (owner, 2026-11-19).
|
||||
4. Logs: the apps' own files may be empty. Verification reads the box OTel collector export
|
||||
(`E:\ApiInstall\OtelCollector\data\logs.jsonl`) over SMB.
|
||||
|
||||
### 0Q.3 Mappings (QA suffix)
|
||||
|
||||
As §4, plus the Deployer interim:
|
||||
|
||||
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|
||||
|---|---|---|---|---|
|
||||
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
|
||||
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
|
||||
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
|
||||
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
|
||||
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
|
||||
|
||||
Remove every GLAuth mapping: HG `SCADA-*` env entries and mxgw `GwAdmin`/`GwReader`. The ScadaBridge `SCADA-*` seed rows
|
||||
stay; they grant nothing in NAM.
|
||||
|
||||
### 0Q.4 The window
|
||||
|
||||
All remote steps are `winrm` deploy scope on vq03/vq04, one app at a time, in the canary order. **Stop at the first
|
||||
`service-bind` warning** (§0.4 stop rule). Each app's code deploy and LDAP change share one restart.
|
||||
|
||||
0. **Before the window (from the Mac, no remote exec):**
|
||||
- stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB;
|
||||
- **add the ScadaBridge role rows with the CLI** while QA still logs in through GLAuth. Log in as `multi-role`,
|
||||
with `--url` pointing at the **active** central. This path is audited and invalidates the cache, unlike the SQL path DEV
|
||||
had to use. Add the five rows from §0Q.3; they grant nothing until the cutover.
|
||||
1. **Phase 0 per box** (`preflight.ps1` from the DEV record, retargeted to the QA layout `E:\ApiInstall\<App>\app`
|
||||
and the service names):
|
||||
- DC reach on 636 and `zimmer-rootca` trust;
|
||||
- back up every service key and every `appsettings*.json` to `E:\ApiInstall\_ldapcut-<ts>\`;
|
||||
- record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa `Secrets:Replication:Enabled`, the
|
||||
OtOpcUa Primary/Standby and ServiceLevels, and the `LdapGroupRoleMapping` overlay rows;
|
||||
- read each app's secrets provider and master-key source from its service env.
|
||||
2. **Seed `nam-ldap-svc-password`** (`seed-secret.ps1`, generalised to read each service's own `Secrets__*` and
|
||||
master-key env):
|
||||
- HG and mxgw: per box;
|
||||
- OtOpcUa: per box, or once if replication is on;
|
||||
- ScadaBridge: **once**, into the shared SQL store, using the active Central's provider settings.
|
||||
|
||||
Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted.
|
||||
3. **HistorianGateway**, vq03 then vq04 (independent single nodes; this is the canary):
|
||||
- deploy `f84af29` with HG's own procedure (`HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/` retargeted to
|
||||
QA);
|
||||
- before the start, rewrite the NSSM env: `Ldap__*` → §6, `Dashboard__GroupToRole__*` → §0Q.3, and the cookie per
|
||||
finding 1.
|
||||
|
||||
Verify on each box: bogus refused, `dohertj2` → Administrator, no `system-side` lines. Then the next box.
|
||||
4. **MxAccessGateway**, vq03 then vq04:
|
||||
- deploy `e74918cc` with mxgw's procedure. It will replace `app\appsettings.json`, so apply the edit to the **new**
|
||||
file before the start: §6 block, QA `GroupToRole`, delete `GwAdmin`/`GwReader`, no GLAuth pin;
|
||||
- mind the nssm `START_PENDING` abort in the repo upgrade script (memory note: use the try/catch fix as on vd03).
|
||||
|
||||
Verify per box as in step 3.
|
||||
5. **ScadaBridge pair**: deploy `2e0ce81e` with the QA pair procedure (`deploy-prep/2026-11-15-143/qa.tmpl.ps1` actions
|
||||
minus `migrate`):
|
||||
- stop both Sites, then both Centrals, so the pair goes down together;
|
||||
- swap the binaries and edit **each** Central's `appsettings.Central.json`: §6 block, drop any `Auth` block;
|
||||
- remove `ScadaBridge__Security__Ldap__ServiceAccountPassword` from **both** Central envs;
|
||||
- start the Centrals (vq03 first), then the Sites.
|
||||
|
||||
Verify:
|
||||
- `dohertj2` gets Administrator+Designer+Deployer at `/auth/login`, at `/auth/token` and on the CLI against the
|
||||
active central;
|
||||
- bogus refused;
|
||||
- LocalDb sync connected; Z28062–Z28069 at their Good baseline; `WarmCanary` connected; one active per pair.
|
||||
6. **OtOpcUa Warm pair**: deploy `cdf433a3` with OtOpcUa's QA procedure (the 2026-11-13 `76e73e7d` pack: stop-both /
|
||||
start-both for a binary upgrade, worker isolation `-Reapply`/`-Verify`).
|
||||
- **Before the start**, add to **both** nodes' env: the 18 `Security__Ldap__*` entries from the DEV record (QA suffix
|
||||
in `GroupToRole`), `Security__Auth__DisableLogin` left `false`, and the cookie per finding 1.
|
||||
- Verify:
|
||||
- ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads `BadOutOfService`;
|
||||
- LocalDb sync connected; the script-runner canary Good;
|
||||
- `dohertj2` → Viewer+Designer+Administrator via cookie and `/auth/token`; bogus refused;
|
||||
- the ScadaBridge `WarmCanary` reconnects.
|
||||
7. **Re-point QA automation** from `multi-role` to `dohertj2`: the qa pack scripts, drills, `apply.sh`.
|
||||
8. **Optional:** oswaltk's Viewer-only login (the only Viewer-only member).
|
||||
9. **Record** in `docs/deploy/2026-11-<dd>-qa-nam-ldap/`, plus a deploy record in each repo.
|
||||
|
||||
**Outages:** HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on
|
||||
the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is
|
||||
also a binary upgrade, and the pairs can't roll those.
|
||||
|
||||
**Rollback:** per app, restore the `_ldapcut-<ts>` keys and files. Binaries roll back with each repo's own procedure
|
||||
(no app here has a schema change). Keep the ScadaBridge `UG_WW_*` rows. The box-local GLAuth stays running until the
|
||||
soak ends.
|
||||
|
||||
**Goal:** move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth
|
||||
test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair.
|
||||
|
||||
Reference in New Issue
Block a user