docs(plans): VD03 NAM AD LDAP cutover plan (not executed)
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,253 @@
|
||||
# VD03 NAM AD LDAP cutover — plan (NOT executed)
|
||||
|
||||
**Written 2026-08-13. Status: PLAN ONLY.** Swap every auth-bearing service on
|
||||
`wonder-app-vd03` from its current login posture (mostly login-disabled) to the real
|
||||
Zimmer NAM Active Directory, with the service-account password held as a
|
||||
`${secret:}` reference in each app's encrypted Secrets store — never in config.
|
||||
|
||||
Grounded in: [`ldap_setup.md`](../../ldap_setup.md) (directory facts + the 2026-08-13
|
||||
end-to-end verification via the HistorianGateway docker container),
|
||||
[`ldap_mapping.md`](../../ldap_mapping.md) (the `UG_WW_*` → role mapping, §4),
|
||||
and the wonder deploy records (memory: `wonder-upgrade-program`).
|
||||
|
||||
## Scope
|
||||
|
||||
| Service on VD03 | In scope | LDAP config surface | Login state today |
|
||||
|---|---|---|---|
|
||||
| OtOpcUa AdminUI | ✅ | `Security:Ldap` (+ `GroupToRole` subkey, merged with a DB overlay) | LDAP enabled; target = TBD in preflight (data plane is API-key, **not** LDAP — untouched) |
|
||||
| ScadaBridge (central UI) | ✅ | `ScadaBridge:Security:Ldap` in `appsettings.Central.json`; roles in the **`LdapGroupMapping` DB table**, not config | **Login DISABLED** (`DisableLogin=true` + `AllowOutsideDevelopment=true` — the two-key set) |
|
||||
| MxAccessGateway dashboard | ✅ | `MxGateway:Ldap` + `MxGateway:Dashboard:GroupToRole` (gRPC stays API-key, untouched) | LDAP enabled per 08-12 notes; target = TBD in preflight |
|
||||
| HistorianGateway dashboard | ✅ | `Ldap` (root section) + `Dashboard:GroupToRole` (gRPC stays API-key, untouched) | **Login DISABLED** (`Dashboard:DisableLogin=true`, deliberately open) |
|
||||
| Overview | ❌ exempt | — | No login **by design** (holds no credentials; references no Auth package) |
|
||||
|
||||
Out of scope: windev's OtOpcUa/mxgw instances, the QA pair (VQ03/VQ04 — do it as a
|
||||
follow-on once VD03 has soaked; QA is where the 2-node/failover behaviour actually gets
|
||||
exercised), and OtOpcUa's OPC UA data plane (API-key auth since Gitea #520).
|
||||
|
||||
## The target LDAP block (identical shape in all four apps)
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"Enabled": true,
|
||||
"Server": "uswap-vdc-nam1.nam.zimmer.com",
|
||||
"Port": 636,
|
||||
"Transport": "Ldaps",
|
||||
"AllowInsecure": false, // see TLS decision below
|
||||
"FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ],
|
||||
"SearchBase": "DC=nam,DC=zimmer,DC=com",
|
||||
"ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
|
||||
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
|
||||
"UserNameAttribute": "sAMAccountName",
|
||||
"DisplayNameAttribute": "name",
|
||||
"GroupAttribute": "memberOf"
|
||||
}
|
||||
```
|
||||
|
||||
Facts already proven live (2026-08-13, HG docker → real NAM AD): the UPN bind as
|
||||
`svc_igntadmin` works (**not** `svc_igntnapp` — that is Ignition's account), the
|
||||
domain-root SearchBase works **on Auth 0.2.1 only**, `memberOf` DNs collapse to bare
|
||||
CNs so `GroupToRole` keys are plain `UG_WW_*` names, and `FallbackServers` walks both
|
||||
DCs on system-side failures. Users log in with the **bare sAMAccountName**.
|
||||
|
||||
## Hard prerequisite: every deployed build must carry Auth 0.2.1
|
||||
|
||||
**None of the four deployed builds has it** (pins landed 2026-08-13, after the last
|
||||
deploy round): OtOpcUa runs `c51c5998`, ScadaBridge `20f2ae05`, mxgw the 08-12 build
|
||||
(Auth 0.2.0), HG `e9c31da` (Auth **0.1.5**). On anything below 0.2.1 a domain-root
|
||||
SearchBase fails **every** login as `ServiceAccountBindFailed` (AD continuation
|
||||
referrals), and on 0.1.5 there is no `FallbackServers` at all. Narrowing SearchBase
|
||||
below the root is NOT an acceptable workaround — user objects span multiple OUs.
|
||||
|
||||
So **Phase 1 is a redeploy of all four apps at their current default-branch tips**
|
||||
(each already pins 0.2.1 and is pushed: OtOpcUa `08e33fc7`, ScadaBridge `d28824d9`,
|
||||
mxgw `1cb14d2`, HG `e42ff31`). This folds into the standing wonder-upgrade practice —
|
||||
HG is behind `main` anyway. Run the standard per-app preflight (migration-delta check,
|
||||
appsettings SHA capture, env backup) from the `wonder-upgrade-program` memory.
|
||||
|
||||
## Decisions to confirm before executing
|
||||
|
||||
1. **Service account.** Use `svc_igntadmin` (proven), or request a dedicated
|
||||
`svc_zbmomww` via ServiceNow first (the `UG_WW_*` groups are ticket-provisioned, so
|
||||
the process exists). Reusing the Ignition admin account couples our uptime to their
|
||||
rotations. **Recommended: start with `svc_igntadmin`, file the dedicated-account
|
||||
ticket in parallel, swap later by rotating one secret per store.**
|
||||
2. **Enabling login on services that are deliberately open.** ScadaBridge and HG run
|
||||
login-disabled on this box today, on purpose. This cutover flips both to
|
||||
authenticated. Confirm that is wanted on the DEV box (it is the point of the
|
||||
exercise, but it changes who can reach those dashboards).
|
||||
3. **Who gets in: VD03 = DEV tier ⇒ map only `_DEV` groups** (ldap_mapping §3 — a
|
||||
deployment maps only its own tier; mapping PROD groups here would make every DEV
|
||||
grant a cross-tier grant). Verify the intended users are actually members:
|
||||
`UG_WW_APP_ADMIN_DEV` (7), `UG_WW_HIST_ADMIN_DEV` (7), `UG_WW_APP_IDE_DEV` (6),
|
||||
`UG_WW_APP_IDE_RO_DEV` (2). If someone who needs access is in none, that is a
|
||||
ServiceNow group-membership request **before** cutover, not a mapping widen.
|
||||
(`BTMGR_RO` has no DEV group — omit.) Check with
|
||||
`ldapprobe --insecure --svc svc_igntadmin@nam.zimmer.com --self-bind <sAMAccountName>`.
|
||||
4. **TLS.** Try `AllowInsecure: false` first — the box is on the corporate network and
|
||||
Windows machine stores on managed boxes usually carry `zimmer-rootca` (installing it
|
||||
is sufficient; the only chain error observed is `UntrustedRoot`, and the RSA-1024
|
||||
hazard is OpenSSL-container-specific, not Windows). If the preflight cert check
|
||||
fails: install `zimmer-rootca` into LocalMachine\Root (preferred) or fall back to
|
||||
`AllowInsecure: true` (DEV-acceptable; transport still encrypted).
|
||||
5. **ScadaBridge plant-floor/operator rows** (`*_OPERATOR`, `SITE_ADMIN_*`,
|
||||
`SUPERVISOR`/`ENGINEER`): ldap_mapping flags these ⚠ DECIDE and they are not
|
||||
env-scoped. **Recommended: leave them out of this cutover** — seed only the
|
||||
platform rows (below) and add operator rows when a real site use-case lands.
|
||||
|
||||
## Phase 0 — preflight (read-only, no changes)
|
||||
|
||||
From the box (servecli :2222; note servecli never survives a reboot — restart it first
|
||||
if dead):
|
||||
|
||||
- **Network:** `Test-NetConnection uswap-vdc-nam1.nam.zimmer.com -Port 636` (and nam2).
|
||||
- **TLS trust:** from PowerShell, open an `SslStream` to nam1:636 (or
|
||||
`certutil -verify` the exported chain) to learn whether `zimmer-rootca` is trusted →
|
||||
settles decision 4.
|
||||
- **Service-account credential, once, without lockout risk** (bad binds count on both
|
||||
DCs per app attempt; policy is 5/15 min): a single bind test from the Mac —
|
||||
`LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D
|
||||
svc_igntadmin@nam.zimmer.com -W -b "DC=nam,DC=zimmer,DC=com" -s base dn`.
|
||||
- **Capture current state per app:** SHA the live appsettings files; record the NSSM
|
||||
registry env (`Parameters\AppEnvironmentExtra` for mxgw/HG; the OtOpcUa/ScadaBridge
|
||||
service envs) — **check no `Ldap__*` keys lurk in service env, which would silently
|
||||
override the JSON edits**; record current `DisableLogin` posture.
|
||||
- **Secrets plumbing:** confirm the four stores + file-based KEKs are healthy (they
|
||||
were authored + round-trip-verified 2026-08-12; KEK backups at
|
||||
`E:\ApiInstall\_kek-backup-20260812T070057`). Stores live outside content roots
|
||||
(OtOpcUa `E:\ApiInstall\OtOpcUa-data\`, mxgw + HG under their `data\`,
|
||||
ScadaBridge per its overlays).
|
||||
|
||||
## Phase 1 — redeploy all four apps at Auth-0.2.1 tips
|
||||
|
||||
Standard wonder deploy recipe per app (memory: `wonder-app-vd03-mxgateway-deploy` +
|
||||
per-app records). Self-contained win-x64 publishes (box has no .NET 10 runtime).
|
||||
Expected deltas are small (each app's tip is its 08-12 deployed SHA + the 0.2.1 pin
|
||||
± UI-sweep commits for HG); still run the migration-delta check — HG's jump
|
||||
(`e9c31da → e42ff31`) carries the Theme 0.4.1/Health 0.3.0/Secrets 0.6.2 sweep, so
|
||||
give it the full preflight (its Secrets 0.6.2 store-path guard will validate the
|
||||
box's paths — they were fixed 08-12, so this should pass, but it is the guard that
|
||||
bit OtOpcUa last round).
|
||||
|
||||
**Gate:** all services healthy on their current (pre-LDAP-swap) config before touching
|
||||
auth. Do not combine the redeploy and the auth swap in one restart — two variables.
|
||||
|
||||
## Phase 2 — seed the secret into each app's store
|
||||
|
||||
Secret name: **`nam-ldap-svc-password`** (same name in every store, so the config
|
||||
reference is identical everywhere and rotation is a mechanical sweep).
|
||||
|
||||
Use the on-box CLI `E:\ApiInstall\_tools\secret` (0.6.2, self-contained). **Trap
|
||||
(proven 08-12): bare CLI verbs read the CLI's own env, NOT the target app's config** —
|
||||
target each store per-invocation:
|
||||
|
||||
```powershell
|
||||
$env:Secrets__SqlitePath = "<that app's store path>"
|
||||
$env:Secrets__MasterKey__Source = "File"
|
||||
$env:Secrets__MasterKey__FilePath = "<that app's KEK file>"
|
||||
E:\ApiInstall\_tools\secret\secret.exe set nam-ldap-svc-password # prompts; never a literal arg
|
||||
```
|
||||
|
||||
Stores to seed (verify paths from the live NSSM/registry env captured in Phase 0):
|
||||
|
||||
| Store | Needed by |
|
||||
|---|---|
|
||||
| OtOpcUa (`E:\ApiInstall\OtOpcUa-data\...`) | AdminUI LDAP login |
|
||||
| ScadaBridge **central** | central UI LDAP login (the `${secret:}` ref lives in `appsettings.Central.json`) |
|
||||
| ScadaBridge **site** | not strictly needed (site does no LDAP) — seed anyway for symmetry; the two stores share one KEK but are separate stores |
|
||||
| MxGateway | dashboard LDAP login |
|
||||
| HistorianGateway | dashboard LDAP login |
|
||||
|
||||
Verify each with `secret.exe get nam-ldap-svc-password` (value round-trip) under the
|
||||
same per-invocation targeting. **Rotation note for the runbook:** one password now
|
||||
lives in 4–5 stores; a rotation is N `set` operations plus zero config changes.
|
||||
|
||||
## Phase 3 — config edits (one app at a time, restart, verify, then next)
|
||||
|
||||
Order: **HG → mxgw → OtOpcUa → ScadaBridge** (proven-pattern first, DB-mapping app
|
||||
last). Back up every file edited; edits go in the box's live overlay files, and the
|
||||
next deploy must re-apply them (add to the deploy-notes convention — the deploy
|
||||
process restores box appsettings, so once these edits ARE the box appsettings they
|
||||
survive; the risk is a fresh-install overwrite, same as every other box-local key).
|
||||
|
||||
### 3a. HistorianGateway
|
||||
- `Ldap` section ← the target block above.
|
||||
- `Dashboard:DisableLogin` → `false`; `Dashboard:GroupToRole` ←
|
||||
`UG_WW_HIST_ADMIN_DEV: Administrator`, `UG_WW_APP_ADMIN_DEV: Administrator`,
|
||||
`UG_WW_APP_IDE_RO_DEV: Viewer`.
|
||||
- Restart `ZbHistorianGateway`; run the Phase 4 checks.
|
||||
|
||||
### 3b. MxAccessGateway
|
||||
- `MxGateway:Ldap` ← target block.
|
||||
- `MxGateway:Dashboard:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`,
|
||||
`UG_WW_APP_IDE_DEV: Viewer`, `UG_WW_APP_IDE_RO_DEV: Viewer` (only two roles exist;
|
||||
Designer collapses to Viewer per ldap_mapping §4).
|
||||
- Restart; verify. (gRPC/API-key surface untouched — do NOT disturb the restored
|
||||
API-key store in `data\`.)
|
||||
|
||||
### 3c. OtOpcUa (AdminUI only)
|
||||
- `Security:Ldap` ← target block (keep any existing `GroupToRole` DB-overlay rows in
|
||||
mind — config merges with the DB overlay; check the RoleGrants page after).
|
||||
- `Security:Ldap:GroupToRole` ← `UG_WW_APP_ADMIN_DEV: Administrator`,
|
||||
`UG_WW_APP_IDE_DEV: Designer`, `UG_WW_APP_IDE_RO_DEV: Viewer`,
|
||||
`UG_WW_HIST_ADMIN_DEV: Viewer` (historization targets are authored here — see
|
||||
ldap_mapping §4 rationale).
|
||||
- Restart the AdminUI/central service. Driver-only nodes are unaffected (no LDAP).
|
||||
- The OPC UA data plane keeps its `otua_` API keys — no consumer impact.
|
||||
|
||||
### 3d. ScadaBridge (central)
|
||||
- `appsettings.Central.json`: `ScadaBridge:Security:Ldap` ← target block; **remove the
|
||||
two disable-login keys** (`DisableLogin` + `AllowOutsideDevelopment`) — note the trap:
|
||||
removing only `AllowOutsideDevelopment` while leaving `DisableLogin=true`
|
||||
**crash-loops** (`DisableLoginGuard`); remove both.
|
||||
- Role rows in the central DB (`LdapGroupMapping` — the only app whose mapping is data,
|
||||
and the only one that can grant two roles per group):
|
||||
```sql
|
||||
INSERT INTO LdapGroupMapping (LdapGroupName, Role) VALUES
|
||||
('UG_WW_APP_ADMIN_DEV', 'Administrator'),
|
||||
('UG_WW_APP_IDE_DEV', 'Designer'),
|
||||
('UG_WW_APP_IDE_DEV', 'Deployer'), -- two rows on purpose (§2.1)
|
||||
('UG_WW_APP_IDE_RO_DEV', 'Viewer');
|
||||
```
|
||||
(Operator/site rows deliberately deferred — decision 5.)
|
||||
- Restart central; site service untouched.
|
||||
|
||||
## Phase 4 — verification (per app, immediately after its restart)
|
||||
|
||||
The log **failure stage** is the diagnostic (proven live, recorded in ldap_setup.md):
|
||||
|
||||
1. **Negative control first** (bogus user + bogus password): expect a clean
|
||||
"invalid/not authorized" with **zero** `WRN ... failed system-side` lines. A
|
||||
`service-bind`-stage WRN = wrong secret value; a `search`-stage WRN = empty
|
||||
password (the `${secret:}` ref did not resolve — check the store/KEK targeting);
|
||||
connect-stage = network/TLS.
|
||||
2. **Real login** with an account in a mapped `_DEV` group → correct role visible in
|
||||
the UI. And one with **no** mapped group → denied (fail-closed zero-group rule).
|
||||
3. **`${secret:}` positive control:** the value must come from the store, not residue —
|
||||
e.g. confirm no `Ldap__ServiceAccountPassword` in service env, and that
|
||||
`secret.exe get` matches what auth uses (a wrong-value canary set+revert is the
|
||||
strong form).
|
||||
4. **Failover spot-check (once, on HG):** block/blackhole nam1:636 briefly →
|
||||
login still works, WRN for nam1 + `sticky preference moved` to nam2; restore.
|
||||
Do not drill on every app — it is the same library and it doubles bad-bind
|
||||
exposure against lockout policy if anything is misconfigured.
|
||||
5. Mind the lockout arithmetic throughout: every failed app attempt = a failed bind
|
||||
on **both** DCs (endpoint walk). Stop and diagnose after 2 failures, don't retry.
|
||||
|
||||
## Rollback
|
||||
|
||||
Per app, in reverse: restore the backed-up appsettings (which restores the previous
|
||||
login posture, including ScadaBridge's two-key disable and HG's `DisableLogin=true`),
|
||||
restart. The seeded secrets are inert once nothing references them — leave them.
|
||||
ScadaBridge: `DELETE FROM LdapGroupMapping WHERE LdapGroupName LIKE 'UG_WW_%'` only if
|
||||
a clean slate is wanted; unreferenced rows grant nothing while login is disabled.
|
||||
Phase 1's redeploys are independent and should NOT be rolled back for an auth problem.
|
||||
|
||||
## Follow-ups this creates
|
||||
|
||||
- QA pair (VQ03/VQ04) cutover with `_QA` groups — the same plan, plus real
|
||||
failover/pair semantics; QA is where `FallbackServers` + sticky actually matter.
|
||||
- Dedicated service account (decision 1) → rotate `nam-ldap-svc-password` in N stores.
|
||||
- Revert the Mac's HG docker container from its temporary NAM flip back to the
|
||||
glauth-redundant pair (uncommitted working-tree change, independent of this plan).
|
||||
- ldap_mapping.md is still untracked in scadaproj — commit it when its DECIDE items
|
||||
settle; this plan depends on its §4 tables.
|
||||
Reference in New Issue
Block a user