From 206c588608f78639feef94e95d0b8d551f6410bc Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 13 Aug 2026 09:24:17 -0400 Subject: [PATCH] docs(plans): VD03 NAM AD LDAP cutover plan (not executed) --- .../plans/2026-08-13-vd03-nam-ldap-cutover.md | 253 ++++++++++++++++++ 1 file changed, 253 insertions(+) create mode 100644 docs/plans/2026-08-13-vd03-nam-ldap-cutover.md diff --git a/docs/plans/2026-08-13-vd03-nam-ldap-cutover.md b/docs/plans/2026-08-13-vd03-nam-ldap-cutover.md new file mode 100644 index 0000000..f2eab76 --- /dev/null +++ b/docs/plans/2026-08-13-vd03-nam-ldap-cutover.md @@ -0,0 +1,253 @@ +# VD03 NAM AD LDAP cutover — plan (NOT executed) + +**Written 2026-08-13. Status: PLAN ONLY.** Swap every auth-bearing service on +`wonder-app-vd03` from its current login posture (mostly login-disabled) to the real +Zimmer NAM Active Directory, with the service-account password held as a +`${secret:}` reference in each app's encrypted Secrets store — never in config. + +Grounded in: [`ldap_setup.md`](../../ldap_setup.md) (directory facts + the 2026-08-13 +end-to-end verification via the HistorianGateway docker container), +[`ldap_mapping.md`](../../ldap_mapping.md) (the `UG_WW_*` → role mapping, §4), +and the wonder deploy records (memory: `wonder-upgrade-program`). + +## Scope + +| Service on VD03 | In scope | LDAP config surface | Login state today | +|---|---|---|---| +| OtOpcUa AdminUI | ✅ | `Security:Ldap` (+ `GroupToRole` subkey, merged with a DB overlay) | LDAP enabled; target = TBD in preflight (data plane is API-key, **not** LDAP — untouched) | +| ScadaBridge (central UI) | ✅ | `ScadaBridge:Security:Ldap` in `appsettings.Central.json`; roles in the **`LdapGroupMapping` DB table**, not config | **Login DISABLED** (`DisableLogin=true` + `AllowOutsideDevelopment=true` — the two-key set) | +| MxAccessGateway dashboard | ✅ | `MxGateway:Ldap` + `MxGateway:Dashboard:GroupToRole` (gRPC stays API-key, untouched) | LDAP enabled per 08-12 notes; target = TBD in preflight | +| HistorianGateway dashboard | ✅ | `Ldap` (root section) + `Dashboard:GroupToRole` (gRPC stays API-key, untouched) | **Login DISABLED** (`Dashboard:DisableLogin=true`, deliberately open) | +| Overview | ❌ exempt | — | No login **by design** (holds no credentials; references no Auth package) | + +Out of scope: windev's OtOpcUa/mxgw instances, the QA pair (VQ03/VQ04 — do it as a +follow-on once VD03 has soaked; QA is where the 2-node/failover behaviour actually gets +exercised), and OtOpcUa's OPC UA data plane (API-key auth since Gitea #520). + +## The target LDAP block (identical shape in all four apps) + +```jsonc +{ + "Enabled": true, + "Server": "uswap-vdc-nam1.nam.zimmer.com", + "Port": 636, + "Transport": "Ldaps", + "AllowInsecure": false, // see TLS decision below + "FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ], + "SearchBase": "DC=nam,DC=zimmer,DC=com", + "ServiceAccountDn": "svc_igntadmin@nam.zimmer.com", + "ServiceAccountPassword": "${secret:nam-ldap-svc-password}", + "UserNameAttribute": "sAMAccountName", + "DisplayNameAttribute": "name", + "GroupAttribute": "memberOf" +} +``` + +Facts already proven live (2026-08-13, HG docker → real NAM AD): the UPN bind as +`svc_igntadmin` works (**not** `svc_igntnapp` — that is Ignition's account), the +domain-root SearchBase works **on Auth 0.2.1 only**, `memberOf` DNs collapse to bare +CNs so `GroupToRole` keys are plain `UG_WW_*` names, and `FallbackServers` walks both +DCs on system-side failures. Users log in with the **bare sAMAccountName**. + +## Hard prerequisite: every deployed build must carry Auth 0.2.1 + +**None of the four deployed builds has it** (pins landed 2026-08-13, after the last +deploy round): OtOpcUa runs `c51c5998`, ScadaBridge `20f2ae05`, mxgw the 08-12 build +(Auth 0.2.0), HG `e9c31da` (Auth **0.1.5**). On anything below 0.2.1 a domain-root +SearchBase fails **every** login as `ServiceAccountBindFailed` (AD continuation +referrals), and on 0.1.5 there is no `FallbackServers` at all. Narrowing SearchBase +below the root is NOT an acceptable workaround — user objects span multiple OUs. + +So **Phase 1 is a redeploy of all four apps at their current default-branch tips** +(each already pins 0.2.1 and is pushed: OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`, +mxgw `1cb14d2`, HG `e42ff31`). This folds into the standing wonder-upgrade practice — +HG is behind `main` anyway. Run the standard per-app preflight (migration-delta check, +appsettings SHA capture, env backup) from the `wonder-upgrade-program` memory. + +## Decisions to confirm before executing + +1. **Service account.** Use `svc_igntadmin` (proven), or request a dedicated + `svc_zbmomww` via ServiceNow first (the `UG_WW_*` groups are ticket-provisioned, so + the process exists). Reusing the Ignition admin account couples our uptime to their + rotations. **Recommended: start with `svc_igntadmin`, file the dedicated-account + ticket in parallel, swap later by rotating one secret per store.** +2. **Enabling login on services that are deliberately open.** ScadaBridge and HG run + login-disabled on this box today, on purpose. This cutover flips both to + authenticated. Confirm that is wanted on the DEV box (it is the point of the + exercise, but it changes who can reach those dashboards). +3. **Who gets in: VD03 = DEV tier ⇒ map only `_DEV` groups** (ldap_mapping §3 — a + deployment maps only its own tier; mapping PROD groups here would make every DEV + grant a cross-tier grant). Verify the intended users are actually members: + `UG_WW_APP_ADMIN_DEV` (7), `UG_WW_HIST_ADMIN_DEV` (7), `UG_WW_APP_IDE_DEV` (6), + `UG_WW_APP_IDE_RO_DEV` (2). If someone who needs access is in none, that is a + ServiceNow group-membership request **before** cutover, not a mapping widen. + (`BTMGR_RO` has no DEV group — omit.) Check with + `ldapprobe --insecure --svc svc_igntadmin@nam.zimmer.com --self-bind `. +4. **TLS.** Try `AllowInsecure: false` first — the box is on the corporate network and + Windows machine stores on managed boxes usually carry `zimmer-rootca` (installing it + is sufficient; the only chain error observed is `UntrustedRoot`, and the RSA-1024 + hazard is OpenSSL-container-specific, not Windows). If the preflight cert check + fails: install `zimmer-rootca` into LocalMachine\Root (preferred) or fall back to + `AllowInsecure: true` (DEV-acceptable; transport still encrypted). +5. **ScadaBridge plant-floor/operator rows** (`*_OPERATOR`, `SITE_ADMIN_*`, + `SUPERVISOR`/`ENGINEER`): ldap_mapping flags these ⚠ DECIDE and they are not + env-scoped. **Recommended: leave them out of this cutover** — seed only the + platform rows (below) and add operator rows when a real site use-case lands. + +## Phase 0 — preflight (read-only, no changes) + +From the box (servecli :2222; note servecli never survives a reboot — restart it first +if dead): + +- **Network:** `Test-NetConnection uswap-vdc-nam1.nam.zimmer.com -Port 636` (and nam2). +- **TLS trust:** from PowerShell, open an `SslStream` to nam1:636 (or + `certutil -verify` the exported chain) to learn whether `zimmer-rootca` is trusted → + settles decision 4. +- **Service-account credential, once, without lockout risk** (bad binds count on both + DCs per app attempt; policy is 5/15 min): a single bind test from the Mac — + `LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D + svc_igntadmin@nam.zimmer.com -W -b "DC=nam,DC=zimmer,DC=com" -s base dn`. +- **Capture current state per app:** SHA the live appsettings files; record the NSSM + registry env (`Parameters\AppEnvironmentExtra` for mxgw/HG; the OtOpcUa/ScadaBridge + service envs) — **check no `Ldap__*` keys lurk in service env, which would silently + override the JSON edits**; record current `DisableLogin` posture. +- **Secrets plumbing:** confirm the four stores + file-based KEKs are healthy (they + were authored + round-trip-verified 2026-08-12; KEK backups at + `E:\ApiInstall\_kek-backup-20260812T070057`). Stores live outside content roots + (OtOpcUa `E:\ApiInstall\OtOpcUa-data\`, mxgw + HG under their `data\`, + ScadaBridge per its overlays). + +## Phase 1 — redeploy all four apps at Auth-0.2.1 tips + +Standard wonder deploy recipe per app (memory: `wonder-app-vd03-mxgateway-deploy` + +per-app records). Self-contained win-x64 publishes (box has no .NET 10 runtime). +Expected deltas are small (each app's tip is its 08-12 deployed SHA + the 0.2.1 pin +± UI-sweep commits for HG); still run the migration-delta check — HG's jump +(`e9c31da → e42ff31`) carries the Theme 0.4.1/Health 0.3.0/Secrets 0.6.2 sweep, so +give it the full preflight (its Secrets 0.6.2 store-path guard will validate the +box's paths — they were fixed 08-12, so this should pass, but it is the guard that +bit OtOpcUa last round). + +**Gate:** all services healthy on their current (pre-LDAP-swap) config before touching +auth. Do not combine the redeploy and the auth swap in one restart — two variables. + +## Phase 2 — seed the secret into each app's store + +Secret name: **`nam-ldap-svc-password`** (same name in every store, so the config +reference is identical everywhere and rotation is a mechanical sweep). + +Use the on-box CLI `E:\ApiInstall\_tools\secret` (0.6.2, self-contained). **Trap +(proven 08-12): bare CLI verbs read the CLI's own env, NOT the target app's config** — +target each store per-invocation: + +```powershell +$env:Secrets__SqlitePath = "" +$env:Secrets__MasterKey__Source = "File" +$env:Secrets__MasterKey__FilePath = "" +E:\ApiInstall\_tools\secret\secret.exe set nam-ldap-svc-password # prompts; never a literal arg +``` + +Stores to seed (verify paths from the live NSSM/registry env captured in Phase 0): + +| Store | Needed by | +|---|---| +| OtOpcUa (`E:\ApiInstall\OtOpcUa-data\...`) | AdminUI LDAP login | +| ScadaBridge **central** | central UI LDAP login (the `${secret:}` ref lives in `appsettings.Central.json`) | +| ScadaBridge **site** | not strictly needed (site does no LDAP) — seed anyway for symmetry; the two stores share one KEK but are separate stores | +| MxGateway | dashboard LDAP login | +| HistorianGateway | dashboard LDAP login | + +Verify each with `secret.exe get nam-ldap-svc-password` (value round-trip) under the +same per-invocation targeting. **Rotation note for the runbook:** one password now +lives in 4–5 stores; a rotation is N `set` operations plus zero config changes. + +## Phase 3 — config edits (one app at a time, restart, verify, then next) + +Order: **HG → mxgw → OtOpcUa → ScadaBridge** (proven-pattern first, DB-mapping app +last). Back up every file edited; edits go in the box's live overlay files, and the +next deploy must re-apply them (add to the deploy-notes convention — the deploy +process restores box appsettings, so once these edits ARE the box appsettings they +survive; the risk is a fresh-install overwrite, same as every other box-local key). + +### 3a. HistorianGateway +- `Ldap` section ← the target block above. +- `Dashboard:DisableLogin` → `false`; `Dashboard:GroupToRole` ← + `UG_WW_HIST_ADMIN_DEV: Administrator`, `UG_WW_APP_ADMIN_DEV: Administrator`, + `UG_WW_APP_IDE_RO_DEV: Viewer`. +- Restart `ZbHistorianGateway`; run the Phase 4 checks. + +### 3b. MxAccessGateway +- `MxGateway:Ldap` ← target block. +- `MxGateway:Dashboard:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`, + `UG_WW_APP_IDE_DEV: Viewer`, `UG_WW_APP_IDE_RO_DEV: Viewer` (only two roles exist; + Designer collapses to Viewer per ldap_mapping §4). +- Restart; verify. (gRPC/API-key surface untouched — do NOT disturb the restored + API-key store in `data\`.) + +### 3c. OtOpcUa (AdminUI only) +- `Security:Ldap` ← target block (keep any existing `GroupToRole` DB-overlay rows in + mind — config merges with the DB overlay; check the RoleGrants page after). +- `Security:Ldap:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`, + `UG_WW_APP_IDE_DEV: Designer`, `UG_WW_APP_IDE_RO_DEV: Viewer`, + `UG_WW_HIST_ADMIN_DEV: Viewer` (historization targets are authored here — see + ldap_mapping §4 rationale). +- Restart the AdminUI/central service. Driver-only nodes are unaffected (no LDAP). +- The OPC UA data plane keeps its `otua_` API keys — no consumer impact. + +### 3d. ScadaBridge (central) +- `appsettings.Central.json`: `ScadaBridge:Security:Ldap` ← target block; **remove the + two disable-login keys** (`DisableLogin` + `AllowOutsideDevelopment`) — note the trap: + removing only `AllowOutsideDevelopment` while leaving `DisableLogin=true` + **crash-loops** (`DisableLoginGuard`); remove both. +- Role rows in the central DB (`LdapGroupMapping` — the only app whose mapping is data, + and the only one that can grant two roles per group): + ```sql + INSERT INTO LdapGroupMapping (LdapGroupName, Role) VALUES + ('UG_WW_APP_ADMIN_DEV', 'Administrator'), + ('UG_WW_APP_IDE_DEV', 'Designer'), + ('UG_WW_APP_IDE_DEV', 'Deployer'), -- two rows on purpose (§2.1) + ('UG_WW_APP_IDE_RO_DEV', 'Viewer'); + ``` + (Operator/site rows deliberately deferred — decision 5.) +- Restart central; site service untouched. + +## Phase 4 — verification (per app, immediately after its restart) + +The log **failure stage** is the diagnostic (proven live, recorded in ldap_setup.md): + +1. **Negative control first** (bogus user + bogus password): expect a clean + "invalid/not authorized" with **zero** `WRN ... failed system-side` lines. A + `service-bind`-stage WRN = wrong secret value; a `search`-stage WRN = empty + password (the `${secret:}` ref did not resolve — check the store/KEK targeting); + connect-stage = network/TLS. +2. **Real login** with an account in a mapped `_DEV` group → correct role visible in + the UI. And one with **no** mapped group → denied (fail-closed zero-group rule). +3. **`${secret:}` positive control:** the value must come from the store, not residue — + e.g. confirm no `Ldap__ServiceAccountPassword` in service env, and that + `secret.exe get` matches what auth uses (a wrong-value canary set+revert is the + strong form). +4. **Failover spot-check (once, on HG):** block/blackhole nam1:636 briefly → + login still works, WRN for nam1 + `sticky preference moved` to nam2; restore. + Do not drill on every app — it is the same library and it doubles bad-bind + exposure against lockout policy if anything is misconfigured. +5. Mind the lockout arithmetic throughout: every failed app attempt = a failed bind + on **both** DCs (endpoint walk). Stop and diagnose after 2 failures, don't retry. + +## Rollback + +Per app, in reverse: restore the backed-up appsettings (which restores the previous +login posture, including ScadaBridge's two-key disable and HG's `DisableLogin=true`), +restart. The seeded secrets are inert once nothing references them — leave them. +ScadaBridge: `DELETE FROM LdapGroupMapping WHERE LdapGroupName LIKE 'UG_WW_%'` only if +a clean slate is wanted; unreferenced rows grant nothing while login is disabled. +Phase 1's redeploys are independent and should NOT be rolled back for an auth problem. + +## Follow-ups this creates + +- QA pair (VQ03/VQ04) cutover with `_QA` groups — the same plan, plus real + failover/pair semantics; QA is where `FallbackServers` + sticky actually matter. +- Dedicated service account (decision 1) → rotate `nam-ldap-svc-password` in N stores. +- Revert the Mac's HG docker container from its temporary NAM flip back to the + glauth-redundant pair (uncommitted working-tree change, independent of this plan). +- ldap_mapping.md is still untracked in scadaproj — commit it when its DECIDE items + settle; this plan depends on its §4 tables.