scadaproj/docs/plans/2026-08-13-vd03-nam-ldap-cutover.md

VD03 NAM AD LDAP cutover — plan (NOT executed)

Written 2026-08-13. Status: PLAN ONLY. Swap every auth-bearing service on wonder-app-vd03 from its current login posture (mostly login-disabled) to the real Zimmer NAM Active Directory, with the service-account password held as a ${secret:} reference in each app's encrypted Secrets store — never in config.

Grounded in: ldap_setup.md (directory facts + the 2026-08-13 end-to-end verification via the HistorianGateway docker container), ldap_mapping.md (the UG_WW_* → role mapping, §4), and the wonder deploy records (memory: wonder-upgrade-program).

Scope

Service on VD03	In scope	LDAP config surface	Login state today
OtOpcUa AdminUI	✅	Security:Ldap (+ GroupToRole subkey, merged with a DB overlay)	LDAP enabled; target = TBD in preflight (data plane is API-key, not LDAP — untouched)
ScadaBridge (central UI)	✅	ScadaBridge:Security:Ldap in appsettings.Central.json; roles in the LdapGroupMapping DB table, not config	Login DISABLED (DisableLogin=true + AllowOutsideDevelopment=true — the two-key set)
MxAccessGateway dashboard	✅	MxGateway:Ldap + MxGateway:Dashboard:GroupToRole (gRPC stays API-key, untouched)	LDAP enabled per 08-12 notes; target = TBD in preflight
HistorianGateway dashboard	✅	Ldap (root section) + Dashboard:GroupToRole (gRPC stays API-key, untouched)	Login DISABLED (Dashboard:DisableLogin=true, deliberately open)
Overview	❌ exempt	—	No login by design (holds no credentials; references no Auth package)

Out of scope: windev's OtOpcUa/mxgw instances, the QA pair (VQ03/VQ04 — do it as a follow-on once VD03 has soaked; QA is where the 2-node/failover behaviour actually gets exercised), and OtOpcUa's OPC UA data plane (API-key auth since Gitea #520).

The target LDAP block (identical shape in all four apps)

{
  "Enabled": true,
  "Server": "uswap-vdc-nam1.nam.zimmer.com",
  "Port": 636,
  "Transport": "Ldaps",
  "AllowInsecure": false,               // see TLS decision below
  "FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ],
  "SearchBase": "DC=nam,DC=zimmer,DC=com",
  "ServiceAccountDn": "svc_igntadmin@nam.zimmer.com",
  "ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
  "UserNameAttribute": "sAMAccountName",
  "DisplayNameAttribute": "name",
  "GroupAttribute": "memberOf"
}

Facts already proven live (2026-08-13, HG docker → real NAM AD): the UPN bind as svc_igntadmin works (not svc_igntnapp — that is Ignition's account), the domain-root SearchBase works on Auth 0.2.1 only, memberOf DNs collapse to bare CNs so GroupToRole keys are plain UG_WW_* names, and FallbackServers walks both DCs on system-side failures. Users log in with the bare sAMAccountName.

Hard prerequisite: every deployed build must carry Auth 0.2.1

None of the four deployed builds has it (pins landed 2026-08-13, after the last deploy round): OtOpcUa runs c51c5998, ScadaBridge 20f2ae05, mxgw the 08-12 build (Auth 0.2.0), HG e9c31da (Auth 0.1.5). On anything below 0.2.1 a domain-root SearchBase fails every login as ServiceAccountBindFailed (AD continuation referrals), and on 0.1.5 there is no FallbackServers at all. Narrowing SearchBase below the root is NOT an acceptable workaround — user objects span multiple OUs.

So Phase 1 is a redeploy of all four apps at their current default-branch tips (each already pins 0.2.1 and is pushed: OtOpcUa 08e33fc7, ScadaBridge d28824d9, mxgw 1cb14d2, HG e42ff31). This folds into the standing wonder-upgrade practice — HG is behind main anyway. Run the standard per-app preflight (migration-delta check, appsettings SHA capture, env backup) from the wonder-upgrade-program memory.

Decisions to confirm before executing

1. Service account. Use svc_igntadmin (proven), or request a dedicated svc_zbmomww via ServiceNow first (the UG_WW_* groups are ticket-provisioned, so the process exists). Reusing the Ignition admin account couples our uptime to their rotations. Recommended: start with svc_igntadmin, file the dedicated-account ticket in parallel, swap later by rotating one secret per store.
2. Enabling login on services that are deliberately open. ScadaBridge and HG run login-disabled on this box today, on purpose. This cutover flips both to authenticated. Confirm that is wanted on the DEV box (it is the point of the exercise, but it changes who can reach those dashboards).
3. Who gets in: VD03 = DEV tier ⇒ map only _DEV groups (ldap_mapping §3 — a deployment maps only its own tier; mapping PROD groups here would make every DEV grant a cross-tier grant). Verify the intended users are actually members: UG_WW_APP_ADMIN_DEV (7), UG_WW_HIST_ADMIN_DEV (7), UG_WW_APP_IDE_DEV (6), UG_WW_APP_IDE_RO_DEV (2). If someone who needs access is in none, that is a ServiceNow group-membership request before cutover, not a mapping widen. (BTMGR_RO has no DEV group — omit.) Check with ldapprobe --insecure --svc svc_igntadmin@nam.zimmer.com --self-bind <sAMAccountName>.
4. TLS. Try AllowInsecure: false first — the box is on the corporate network and Windows machine stores on managed boxes usually carry zimmer-rootca (installing it is sufficient; the only chain error observed is UntrustedRoot, and the RSA-1024 hazard is OpenSSL-container-specific, not Windows). If the preflight cert check fails: install zimmer-rootca into LocalMachine\Root (preferred) or fall back to AllowInsecure: true (DEV-acceptable; transport still encrypted).
5. ScadaBridge plant-floor/operator rows (*_OPERATOR, SITE_ADMIN_*, SUPERVISOR/ENGINEER): ldap_mapping flags these ⚠ DECIDE and they are not env-scoped. Recommended: leave them out of this cutover — seed only the platform rows (below) and add operator rows when a real site use-case lands.

Phase 0 — preflight (read-only, no changes)

From the box (servecli :2222; note servecli never survives a reboot — restart it first if dead):

• Network: Test-NetConnection uswap-vdc-nam1.nam.zimmer.com -Port 636 (and nam2).
• TLS trust: from PowerShell, open an SslStream to nam1:636 (or certutil -verify the exported chain) to learn whether zimmer-rootca is trusted → settles decision 4.
• Service-account credential, once, without lockout risk (bad binds count on both DCs per app attempt; policy is 5/15 min): a single bind test from the Mac — LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D svc_igntadmin@nam.zimmer.com -W -b "DC=nam,DC=zimmer,DC=com" -s base dn.
• Capture current state per app: SHA the live appsettings files; record the NSSM registry env (Parameters\AppEnvironmentExtra for mxgw/HG; the OtOpcUa/ScadaBridge service envs) — check no Ldap__* keys lurk in service env, which would silently override the JSON edits; record current DisableLogin posture.
• Secrets plumbing: confirm the four stores + file-based KEKs are healthy (they were authored + round-trip-verified 2026-08-12; KEK backups at E:\ApiInstall\_kek-backup-20260812T070057). Stores live outside content roots (OtOpcUa E:\ApiInstall\OtOpcUa-data\, mxgw + HG under their data\, ScadaBridge per its overlays).

Phase 1 — redeploy all four apps at Auth-0.2.1 tips

Standard wonder deploy recipe per app (memory: wonder-app-vd03-mxgateway-deploy + per-app records). Self-contained win-x64 publishes (box has no .NET 10 runtime). Expected deltas are small (each app's tip is its 08-12 deployed SHA + the 0.2.1 pin ± UI-sweep commits for HG); still run the migration-delta check — HG's jump (e9c31da → e42ff31) carries the Theme 0.4.1/Health 0.3.0/Secrets 0.6.2 sweep, so give it the full preflight (its Secrets 0.6.2 store-path guard will validate the box's paths — they were fixed 08-12, so this should pass, but it is the guard that bit OtOpcUa last round).

Gate: all services healthy on their current (pre-LDAP-swap) config before touching auth. Do not combine the redeploy and the auth swap in one restart — two variables.

Phase 2 — seed the secret into each app's store

Secret name: nam-ldap-svc-password (same name in every store, so the config reference is identical everywhere and rotation is a mechanical sweep).

Use the on-box CLI E:\ApiInstall\_tools\secret (0.6.2, self-contained). Trap (proven 08-12): bare CLI verbs read the CLI's own env, NOT the target app's config — target each store per-invocation:

$env:Secrets__SqlitePath = "<that app's store path>"
$env:Secrets__MasterKey__Source = "File"
$env:Secrets__MasterKey__FilePath = "<that app's KEK file>"
E:\ApiInstall\_tools\secret\secret.exe set nam-ldap-svc-password   # prompts; never a literal arg

Stores to seed (verify paths from the live NSSM/registry env captured in Phase 0):

Store	Needed by
OtOpcUa (E:\ApiInstall\OtOpcUa-data\...)	AdminUI LDAP login
ScadaBridge central	central UI LDAP login (the ${secret:} ref lives in appsettings.Central.json)
ScadaBridge site	not strictly needed (site does no LDAP) — seed anyway for symmetry; the two stores share one KEK but are separate stores
MxGateway	dashboard LDAP login
HistorianGateway	dashboard LDAP login

Verify each with secret.exe get nam-ldap-svc-password (value round-trip) under the same per-invocation targeting. Rotation note for the runbook: one password now lives in 4–5 stores; a rotation is N set operations plus zero config changes.

Phase 3 — config edits (one app at a time, restart, verify, then next)

Order: HG → mxgw → OtOpcUa → ScadaBridge (proven-pattern first, DB-mapping app last). Back up every file edited; edits go in the box's live overlay files, and the next deploy must re-apply them (add to the deploy-notes convention — the deploy process restores box appsettings, so once these edits ARE the box appsettings they survive; the risk is a fresh-install overwrite, same as every other box-local key).

3a. HistorianGateway

• Ldap section ← the target block above.
• Dashboard:DisableLogin → false; Dashboard:GroupToRole ← UG_WW_HIST_ADMIN_DEV: Administrator, UG_WW_APP_ADMIN_DEV: Administrator, UG_WW_APP_IDE_RO_DEV: Viewer.
• Restart ZbHistorianGateway; run the Phase 4 checks.

3b. MxAccessGateway

• MxGateway:Ldap ← target block.
• MxGateway:Dashboard:GroupToRole ← UG_WW_APP_ADMIN_DEV: Administrator, UG_WW_APP_IDE_DEV: Viewer, UG_WW_APP_IDE_RO_DEV: Viewer (only two roles exist; Designer collapses to Viewer per ldap_mapping §4).
• Restart; verify. (gRPC/API-key surface untouched — do NOT disturb the restored API-key store in data\.)

3c. OtOpcUa (AdminUI only)

• Security:Ldap ← target block (keep any existing GroupToRole DB-overlay rows in mind — config merges with the DB overlay; check the RoleGrants page after).
• Security:Ldap:GroupToRole ← UG_WW_APP_ADMIN_DEV: Administrator, UG_WW_APP_IDE_DEV: Designer, UG_WW_APP_IDE_RO_DEV: Viewer, UG_WW_HIST_ADMIN_DEV: Viewer (historization targets are authored here — see ldap_mapping §4 rationale).
• Restart the AdminUI/central service. Driver-only nodes are unaffected (no LDAP).
• The OPC UA data plane keeps its otua_ API keys — no consumer impact.

3d. ScadaBridge (central)

• appsettings.Central.json: ScadaBridge:Security:Ldap ← target block; remove the two disable-login keys (DisableLogin + AllowOutsideDevelopment) — note the trap: removing only AllowOutsideDevelopment while leaving DisableLogin=true crash-loops (DisableLoginGuard); remove both.
• Role rows in the central DB (LdapGroupMapping — the only app whose mapping is data, and the only one that can grant two roles per group):

  INSERT INTO LdapGroupMapping (LdapGroupName, Role) VALUES
    ('UG_WW_APP_ADMIN_DEV',  'Administrator'),
    ('UG_WW_APP_IDE_DEV',    'Designer'),
    ('UG_WW_APP_IDE_DEV',    'Deployer'),     -- two rows on purpose (§2.1)
    ('UG_WW_APP_IDE_RO_DEV', 'Viewer');
  

  (Operator/site rows deliberately deferred — decision 5.)
• Restart central; site service untouched.

Phase 4 — verification (per app, immediately after its restart)

The log failure stage is the diagnostic (proven live, recorded in ldap_setup.md):

1. Negative control first (bogus user + bogus password): expect a clean "invalid/not authorized" with zero WRN ... failed system-side lines. A service-bind-stage WRN = wrong secret value; a search-stage WRN = empty password (the ${secret:} ref did not resolve — check the store/KEK targeting); connect-stage = network/TLS.
2. Real login with an account in a mapped _DEV group → correct role visible in the UI. And one with no mapped group → denied (fail-closed zero-group rule).
3. ${secret:} positive control: the value must come from the store, not residue — e.g. confirm no Ldap__ServiceAccountPassword in service env, and that secret.exe get matches what auth uses (a wrong-value canary set+revert is the strong form).
4. Failover spot-check (once, on HG): block/blackhole nam1:636 briefly → login still works, WRN for nam1 + sticky preference moved to nam2; restore. Do not drill on every app — it is the same library and it doubles bad-bind exposure against lockout policy if anything is misconfigured.
5. Mind the lockout arithmetic throughout: every failed app attempt = a failed bind on both DCs (endpoint walk). Stop and diagnose after 2 failures, don't retry.

Rollback

Per app, in reverse: restore the backed-up appsettings (which restores the previous login posture, including ScadaBridge's two-key disable and HG's DisableLogin=true), restart. The seeded secrets are inert once nothing references them — leave them. ScadaBridge: DELETE FROM LdapGroupMapping WHERE LdapGroupName LIKE 'UG_WW_%' only if a clean slate is wanted; unreferenced rows grant nothing while login is disabled. Phase 1's redeploys are independent and should NOT be rolled back for an auth problem.

Follow-ups this creates

• QA pair (VQ03/VQ04) cutover with _QA groups — the same plan, plus real failover/pair semantics; QA is where FallbackServers + sticky actually matter.
• Dedicated service account (decision 1) → rotate nam-ldap-svc-password in N stores.
• Revert the Mac's HG docker container from its temporary NAM flip back to the glauth-redundant pair (uncommitted working-tree change, independent of this plan).
• ldap_mapping.md is still untracked in scadaproj — commit it when its DECIDE items settle; this plan depends on its §4 tables.