Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
66 KiB
NAM AD LDAP cutover — DEV vd03, then QA vq03/vq04 (DEV + QA EXECUTED 2026-11-19 on the interim account)
Status lives in Gitea. Execution is tracked by scadaproj#6. The dedicated, shared LDAP-lookup service account is scadaproj#58, a QA prerequisite (§5, D1). This file is the design and runbook. It supersedes
2026-08-13-vd03-nam-ldap-cutover.md(§10 lists what changed). The independent review of 2026-11-15 is applied (§11).
Written 2026-11-15. Status: PLAN ONLY. Nothing here has been executed.
EXECUTED ON DEV 2026-11-19 (§0): all four apps on vd03 log in against NAM AD. Record:
docs/deploy/2026-11-19-vd03-nam-ldap/. The DC-failover checks
(§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's RequireHttpsCookie
on a plain-HTTP AdminUI (record, finding 1). QA plan: §0Q (written 2026-11-19, not executed; QA hold stands).
Updated 2026-11-19: §0 is the DEV vd03 execution plan using the groups and accounts that exist today, so
it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full
prerequisites in §5–§7, and the QA deploy hold still stands.
0. DEV vd03 now, with today's groups and accounts (2026-11-19)
0.1 What the directory holds today
Re-enumerated 2026-11-19 from the Mac (ldapsearch over LDAPS to nam1, bound as svc_igntadmin; that bind is
the D1 interim's single verification bind).
- Groups: 41
UG_WW_*, the 39 from 08-12 plusUG_WW_PAR_CVD_OperandUG_WW_PAR_CVD_Super. Neither is a platform group, so neither is mapped. There is still noUG_WW_APP_DEPLOY_*group. - Names: the DEV platform names are exactly as in §4. Copy them with this case.
| DEV group | Members (2026-11-19) |
|---|---|
UG_WW_APP_ADMIN_DEV (7) |
chamalas, millerh5, benjamd2, marting4, dohertj2, oswaltk, Luis Bastidas |
UG_WW_APP_IDE_DEV (6) |
chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
UG_WW_APP_IDE_RO_DEV (2) |
benjamd2, oswaltk |
UG_WW_HIST_ADMIN_DEV (7) |
the same 7 people as APP_ADMIN_DEV |
UG_WW_CITRIX_DEV (6) |
never mapped; culverj holds only this group, so they get no role on DEV |
Accounts used on DEV (no new accounts):
| Use | Account | Groups and resulting roles | Credential |
|---|---|---|---|
| LDAP lookup (D1 interim) | svc_igntadmin@nam.zimmer.com |
No UG_WW_* group, so it cannot sign in to any app, which is correct for a lookup account |
Mac Keychain, service btdb-manager/ldap. Seeded into each store as nam-ldap-svc-password |
| Positive login and automation (D3 option b) | dohertj2 |
APP_ADMIN_DEV + APP_IDE_DEV + HIST_ADMIN_DEV. Gets Administrator everywhere, plus Designer in OtOpcUa and ScadaBridge, plus Deployer in ScadaBridge (§0.2) |
Mac Keychain (NAM\dohertj2) |
| Negative control: zero roles | adm_dohertj2 |
3 groups, none UG_WW_*. The library passes it (memberOf is non-empty), the app maps no role, and §3.2 must refuse it |
Mac Keychain (NAM\adm_dohertj2) |
| Negative control: bad credentials | a bogus user/password | — | — |
Viewer is not exercised live on DEV. Only benjamd2 and oswaltk hold APP_IDE_RO_DEV, and both also hold
APP_ADMIN_DEV. The Viewer mapping is checked by reading the effective config back (A4). A real Viewer-only
login waits for QA or for a volunteer.
0.2 Deviations from the full plan (DEV only)
| Full plan | DEV now | Why it's acceptable | Ends when |
|---|---|---|---|
| D1: dedicated account from scadaproj#58 | svc_igntadmin |
The interim §5 already allows on DEV. Tell the Ignition team before §0.4 step 3: a wrong secret here locks their admin account (2 bad binds per login, §1.3) | scadaproj#58 is delivered; swap it in during the soak (§7) |
| D3: a NAM automation account | dohertj2 (option b) |
It holds every DEV role the scripts need | The D3 account exists |
D6: UG_WW_APP_DEPLOY_DEV → Deployer |
UG_WW_HIST_ADMIN_DEV → Deployer, in ScadaBridge only |
Still one group, one role per app (§3.1): ScadaBridge maps HIST_ADMIN to nothing else. Its members are identical to APP_ADMIN_DEV, so the effect is "DEV admins can deploy", the SoD collapse already accepted for this directory (ldap_setup.md). It is a DEV-only row |
D6's group exists: add APP_DEPLOY_DEV → Deployer, then delete the HIST_ADMIN_DEV row |
D4: zimmer-rootca in LocalMachine\Root |
Phase 0 checks it. vd03 is domain-joined, so the root is expected to be there via GPO [I]. If it isn't, run DEV with AllowInsecure=true and record it |
DEV only, short-lived (D4 b) | The root is installed |
| D7: membership confirmed per person | Done for DEV by §0.1 | — | — |
Unchanged: the standards (§3), the LDAP block (§6, with ServiceAccountDn: svc_igntadmin@nam.zimmer.com),
the restart rules, the §7.6 verification and stop rule, and the rollback (§7.7).
0.3 Builds on vd03
| App | On vd03 (2026-11-19) | Carries | Action |
|---|---|---|---|
| HistorianGateway | f84af29 |
#136 (case-insensitive GroupToRole, mapper fault refused) |
none |
| MxAccessGateway | e74918cc |
already refuses zero roles | none |
| OtOpcUa | cdf433a35 |
#913 (§2): options passthrough, zero-role refusal, library logger | none |
| ScadaBridge | 061432c8 |
neither #144 nor #145 | Deploy main 2e0ce81e (§0.4 step 1) |
ScadaBridge 061432c8..2e0ce81e is 11 commits: #144, #145 and the #116 leader-auto-down default (already
drilled on vd03, 2026-11-15). No migration and no wire change. Use the procedure in
docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md (single central + site; take a
DB backup first). With login still off, the dev user holds every role, so #144/#145 have no visible effect
until step 6.
0.4 Order of work on vd03
All remote steps stay inside the winrm deploy scope (§7). One app at a time; each one passes §7.6 before the next.
- ScadaBridge binary deploy
2e0ce81e. Verify with the deploy's normal checks: health, the site connected, the CLI works as the login-off dev user. - Phase 0 on vd03 (§7.0), reduced: DC reach on 636 from vd03; the
zimmer-rootcacheck (decides D4); back up every app's config and env; record OtOpcUaSecurity__Auth__DisableLogin(expectedtrue), mxgw's environment name, itsServerCertificateThumbprintSha256and baseGwAdmin/GwReader; each app's secrets store and KEK source; the OtOpcUaLdapGroupRoleMappingrows. Builds per §0.3. The service-account bind was done 2026-11-19 (§0.1), so don't repeat it. - Seed
nam-ldap-svc-passwordin each store (§7.1): HG, mxgw and ScadaBridge Central through their/admin/secretspages (login is still off on all three), and OtOpcUa's store with thesecretCLI or its page. Verify each read-back value with one LdapProbe bind assvc_igntadmin. - HistorianGateway (the canary), §7.2. §6 + §4.1 DEV mapping,
Dashboard:DisableLogin=false. Run §7.6 fully, including the nam1→nam2 failover check. - MxAccessGateway, §7.3. §6 + §4.2 DEV mapping; delete the base
GwAdmin/GwReader; remove any GLAuth pin;DisableLogin=false. - ScadaBridge, §7.4.
- a. While login is still off, add the role mappings with the CLI (
--urlthe vd03 central):UG_WW_APP_ADMIN_DEV→Administrator,UG_WW_APP_IDE_DEV→Designer,UG_WW_APP_IDE_RO_DEV→Viewer,UG_WW_HIST_ADMIN_DEV→Deployer (DEV interim, §0.2). The §4.4 SQL fallback works too: put theHIST_ADMIN_DEVrow in its@t. - b. Write
ScadaBridge:Security:Ldap(§6). Remove bothDisableLoginandAllowOutsideDevelopment. - c. Restart the central. Re-point scripts (
apply.sh, drill and deploy packs) atdohertj2, and confirm/auth/tokenand the CLI (HTTP Basic) work with it. - d. Extra §7.6 check:
adm_dohertj2is refused at/auth/login, at/auth/tokenand on the management/CLI path (#145).
- a. While login is still off, add the role mappings with the CLI (
- OtOpcUa AdminUI, §7.5. §6 as
Security__Ldap__*withFallbackServers__0(the #913 build honours it), the §4.3 DEVGroupToRole, thenSecurity__Auth__DisableLogin=false. Restart the single node. §7.6 plus acceptance A2 (nam1 blocked → login through nam2, with the library'ssticky preference movedline).adm_dohertj2is refused at/auth/loginand/auth/token. - Record the run as
docs/deploy/2026-11-<dd>-vd03-nam-ldap-cutover.mdin scadaproj, plus a line in each app's deployment notes. The mxgw notes must say "deleteGwAdmin/GwReaderfrom the base file on every redeploy". Tick the DEV items on scadaproj#6.
Stop rule (§7.6 rule 6) for this run: the account is Ignition's, so stop at the first service-bind
warning, not the second. Ask IT for the account's lockout state before any retry.
Soak: a few working days of real logins on DEV.
0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19; EXECUTED 2026-11-19)
Executed 2026-11-19 15:29–16:10 EST: all four apps on both boxes on the DEV builds and NAM AD; record
docs/deploy/2026-11-19-qa-nam-ldap/. Optional step 8 (oswaltk) not run.
Owner decisions (2026-11-19):
- QA binds as
svc_igntadminas well (interim). This supersedes A6 and the §7 note "QA never runs onsvc_igntadmin". Both tiers move to the scadaproj#58 account together when it arrives: re-seednam-ldap-svc-passwordin every store, then a rolling restart per app. - QA gets exactly the builds DEV runs.
- One window: each app's code deploy and its LDAP change share one restart.
Gate: the owner lifts the QA hold and names the window. Tell the Ignition team before the window: QA adds 8 more stores that reference their admin account.
0Q.1 What QA holds today
Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP).
| DEV-proven groups | QA members (2026-11-19) |
|---|---|
UG_WW_APP_ADMIN_QA (6) |
chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
UG_WW_APP_IDE_QA (6) |
the same 6 |
UG_WW_APP_IDE_RO_QA (1) |
oswaltk (Viewer only, so a live Viewer test is possible with them) |
UG_WW_HIST_ADMIN_QA (6) |
the same 6 as APP_ADMIN_QA, so the §0.2 Deployer interim carries over |
UG_WW_BTMGR_RO_QA (0) |
empty; mapped per §4 anyway (grants nobody anything) |
Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP. Anonymous GET /
returns 302 → /login on :5220, :5130, :8085 and :9000. So QA is a directory swap, not a "turn login on". The
GLAuth multi-role automation identity stops working at the cutover; the D3 interim is dohertj2, as on DEV.
| App | QA build now | → target (= DEV) | Where its LDAP config lives (QA) | Notes |
|---|---|---|---|---|
| HistorianGateway (per box) | 6365f34 |
f84af29 (#136) |
NSSM env Ldap__*, Dashboard__GroupToRole__SCADA-* [I] |
app\appsettings.json has Dashboard:RequireHttpsCookie=true, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below) |
| MxAccessGateway (per box) | 76a71bc8 |
e74918cc |
app\appsettings.json: GLAuth localhost:3893, dc=zb,dc=local, ${secret:ldap/mxgateway/bind}, base GwAdmin/GwReader |
Phase 0 records the environment name and any overlay. Delete GwAdmin/GwReader from the base file (§4.2) |
| ScadaBridge (pair) | 061432c8 |
2e0ce81e (#144/#145) |
app\appsettings.Central.json (GLAuth 127.0.0.1:3893, dc=scadabridge) + Central env password |
Secrets:Store:Provider=SqlServer: one shared store, so seed once. RequireHttpsCookie=false already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other |
| OtOpcUa (Warm pair) | 76e73e7d |
cdf433a3 (#913; 83 commits, no migration) |
service env [I] | Warm redundancy (Mode=Warm); Phase 0 records which node is Primary. Needs the cookie decision (finding 1) |
Secrets master keys: the shipped files say Source=Environment (ZB_SECRETS_MASTER_KEY) for HG, mxgw and OtOpcUa. The
service env may override that. The seed script reads each service's own env, so it never assumes.
0Q.2 Findings carried from DEV
- Plain-HTTP cookies. OtOpcUa's auth cookie is
Secureby default, so its AdminUI login loops over HTTP. On DEV the fix wasSecurity__Cookie__RequireHttpsCookie=false. QA OtOpcUa login works today over HTTP, so Phase 0 checks whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HGDashboard:RequireHttpsCookiethe same way. - Secrets CLI. The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store
schema or envelope change), copied to
E:\ApiInstall\_tools\secret-0.6.6\on each box. - Do not add firewall rules or run DC-failover drills (owner, 2026-11-19).
- Logs: the apps' own files may be empty. Verification reads the box OTel collector export
(
E:\ApiInstall\OtelCollector\data\logs.jsonl) over SMB.
0Q.3 Mappings (QA suffix)
As §4, plus the Deployer interim:
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
UG_WW_APP_ADMIN_QA |
Administrator | Administrator | Administrator | Administrator |
UG_WW_APP_IDE_QA |
Designer | Designer | Viewer | — |
UG_WW_APP_IDE_RO_QA |
Viewer | Viewer | Viewer | Viewer |
UG_WW_HIST_ADMIN_QA |
Viewer | Deployer (interim) | — | Administrator |
UG_WW_BTMGR_RO_QA |
— | Viewer | — | Viewer |
Remove every GLAuth mapping: HG SCADA-* env entries and mxgw GwAdmin/GwReader. The ScadaBridge SCADA-* seed rows
stay; they grant nothing in NAM.
0Q.4 The window
All remote steps are winrm deploy scope on vq03/vq04, one app at a time, in the canary order. Stop at the first
service-bind warning (§0.4 stop rule). Each app's code deploy and LDAP change share one restart.
-
Before the window (from the Mac, no remote exec):
- stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB;
- add the ScadaBridge role rows with the CLI while QA still logs in through GLAuth. Log in as
multi-role, with--urlpointing at the active central. This path is audited and invalidates the cache, unlike the SQL path DEV had to use. Add the five rows from §0Q.3; they grant nothing until the cutover.
-
Phase 0 per box (
preflight.ps1from the DEV record, retargeted to the QA layoutE:\ApiInstall\<App>\appand the service names):- DC reach on 636 and
zimmer-rootcatrust; - back up every service key and every
appsettings*.jsontoE:\ApiInstall\_ldapcut-<ts>\; - record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa
Secrets:Replication:Enabled, the OtOpcUa Primary/Standby and ServiceLevels, and theLdapGroupRoleMappingoverlay rows; - read each app's secrets provider and master-key source from its service env.
- DC reach on 636 and
-
Seed
nam-ldap-svc-password(seed-secret.ps1, generalised to read each service's ownSecrets__*and master-key env):- HG and mxgw: per box;
- OtOpcUa: per box, or once if replication is on;
- ScadaBridge: once, into the shared SQL store, using the active Central's provider settings.
Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted.
-
HistorianGateway, vq03 then vq04 (independent single nodes; this is the canary):
- deploy
f84af29with HG's own procedure (HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/retargeted to QA); - before the start, rewrite the NSSM env:
Ldap__*→ §6,Dashboard__GroupToRole__*→ §0Q.3, and the cookie per finding 1.
Verify on each box: bogus refused,
dohertj2→ Administrator, nosystem-sidelines. Then the next box. - deploy
-
MxAccessGateway, vq03 then vq04:
- deploy
e74918ccwith mxgw's procedure. It will replaceapp\appsettings.json, so apply the edit to the new file before the start: §6 block, QAGroupToRole, deleteGwAdmin/GwReader, no GLAuth pin; - mind the nssm
START_PENDINGabort in the repo upgrade script (memory note: use the try/catch fix as on vd03).
Verify per box as in step 3.
- deploy
-
ScadaBridge pair: deploy
2e0ce81ewith the QA pair procedure (deploy-prep/2026-11-15-143/qa.tmpl.ps1actions minusmigrate):- stop both Sites, then both Centrals, so the pair goes down together;
- swap the binaries and edit each Central's
appsettings.Central.json: §6 block, drop anyAuthblock; - remove
ScadaBridge__Security__Ldap__ServiceAccountPasswordfrom both Central envs; - start the Centrals (vq03 first), then the Sites.
Verify:
dohertj2gets Administrator+Designer+Deployer at/auth/login, at/auth/tokenand on the CLI against the active central;- bogus refused;
- LocalDb sync connected; Z28062–Z28069 at their Good baseline;
WarmCanaryconnected; one active per pair.
-
OtOpcUa Warm pair: deploy
cdf433a3with OtOpcUa's QA procedure (the 2026-11-1376e73e7dpack: stop-both / start-both for a binary upgrade, worker isolation-Reapply/-Verify).- Before the start, add to both nodes' env: the 18
Security__Ldap__*entries from the DEV record (QA suffix inGroupToRole),Security__Auth__DisableLoginleftfalse, and the cookie per finding 1. - Verify:
- ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads
BadOutOfService; - LocalDb sync connected; the script-runner canary Good;
dohertj2→ Viewer+Designer+Administrator via cookie and/auth/token; bogus refused;- the ScadaBridge
WarmCanaryreconnects.
- ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads
- Before the start, add to both nodes' env: the 18
-
Re-point QA automation from
multi-roletodohertj2: the qa pack scripts, drills,apply.sh. -
Optional: oswaltk's Viewer-only login (the only Viewer-only member).
-
Record in
docs/deploy/2026-11-<dd>-qa-nam-ldap/, plus a deploy record in each repo.
Outages: HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is also a binary upgrade, and the pairs can't roll those.
Rollback: per app, restore the _ldapcut-<ts> keys and files. Binaries roll back with each repo's own procedure
(no app here has a schema change). Keep the ScadaBridge UG_WW_* rows. The box-local GLAuth stays running until the
soak ends.
Goal: move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair.
- The service-account password lives as a
${secret:}reference in each app's encrypted Secrets store, never in config. - Every app maps AD groups to its roles from one canonical matrix,
ldap_mapping.md, under two family rules (§3):- one AD group grants exactly one role per app;
- a user whose groups map to no role cannot sign in (cookie or token).
In scope: HistorianGateway (dashboard), MxAccessGateway (dashboard), ScadaBridge (Central UI + CLI), OtOpcUa (AdminUI).
Out of scope:
- Overview: no login by design.
- The machine-to-machine surfaces stay API-key based: OtOpcUa's OPC UA data plane, the mxgw and HG gRPC APIs, and ScadaBridge's inbound API. They can still be disrupted by the restarts here (§7.0).
- ignitionoee: on hold (
ldap_mapping.md§6).
Evidence key: [V] = verified in source or a file, with the citation; [I] = inferred, to be confirmed in Phase 0. Source paths are relative to each repo.
1. Current state (2026-11-15)
1.1 Builds and the Auth library
The 2026-08-13 plan's "redeploy to Auth 0.2.1" phase is obsolete:
| App | Deployed on DEV and QA | Auth pin [V] |
|---|---|---|
| OtOpcUa | 76e73e7d |
0.4.1 (Directory.Packages.props:238-240) |
| ScadaBridge | 061432c8 |
0.5.0 (ScadaBridge#46) |
| MxAccessGateway | 76a71bc8 |
0.4.1 |
| HistorianGateway | 6365f34 |
0.4.1 |
Domain-root SearchBase (≥ 0.2.1), FallbackServers (≥ 0.2.0 in the library, but dropped by OtOpcUa,
§2) and BindMode with the empty-password guard (≥ 0.4.1) are all available.
Before the cutover:
- OtOpcUa (§2) and ScadaBridge (§2A) need code changes and a deploy.
- HistorianGateway (§2B) gets an optional hardening change.
1.2 Login state per app and tier
| App | DEV vd03 | QA vq03/vq04 |
|---|---|---|
| HistorianGateway | Login OFF: Dashboard:DisableLogin=true in appsettings.Production.json. LDAP points at the shared GLAuth 10.100.0.35:3893, plain [V, file]. |
Login ON against the box-local GLAuth. Config is in the NSSM env (Ldap__*, GroupToRole SCADA-Admins/SCADA-Designers) [I]. KEK at C:\ProgramData\ZB\kek\historiangw.kek [V, 11-13 evidence]. |
| MxAccessGateway | Login OFF: MxGateway:Dashboard:DisableLogin=true [V, file]. LDAP 127.0.0.1:3893, dc=scadalink,dc=local. The deployed build refuses DisableLogin=true under Production (GatewayOptionsValidator.cs:484-488), so vd03 mxgw is not running as Production [V by inference from it booting]; Phase 0 records the environment name. |
Login ON: localhost:3893, dc=zb,dc=local, ${secret:ldap/mxgateway/bind} [V, file]. ⚠ deployment.md says the box-local GLAuth is dc=scadabridge,dc=local [I]. |
| ScadaBridge | Login OFF: ScadaBridge:Security:Auth:DisableLogin=true + AllowOutsideDevelopment=true [V, file]. With login off, the management/CLI path authenticates every caller as the all-roles dev user (ManagementService/ManagementAuthenticator.cs:63-71: "multi-role", Roles.All) [V]. LDAP localhost:3893. |
Login ON, box-local GLAuth 127.0.0.1:3893 [V]. Every CLI, deploy and drill script logs in as GLAuth multi-role. |
| OtOpcUa (AdminUI) | Code defaults (Enabled=true, Server=localhost, Port=3893, SearchBase=dc=zb,dc=local, ServiceAccountDn=""), plus files holding Transport=Ldaps and DevStubMode=false; no Security__Ldap__* in the env [V, 11-13 record]. Security__Auth__DisableLogin is in the env, value unknown. |
Unknown; Phase 0 reads it. |
1.3 Directory facts
Still valid, from ldap_setup.md:
- DCs:
uswap-vdc-nam1.nam.zimmer.com(10.201.216.2) anduswap-vdc-nam2.nam.zimmer.com(10.201.138.78). - Transports: LDAPS 636, StartTLS and plain 389 all work.
- TLS: the certs are RSA-1024, chain
leaf ← zimmer-issueca1 ← zimmer-rootca. The only chain error isUntrustedRoot, so installing the root is sufficient. Windows accepts the 1024-bit key. - Directory shape:
SearchBaseDC=nam,DC=zimmer,DC=com; login attributesAMAccountName;memberOfcollapses to bare CNs. - Reachability: from the Mac, nam1/nam2 on 389 and 636 are open (2026-11-15). Not proven from the boxes.
Login names. In BindMode=ServiceAccount (this plan), AcceptDomainQualifiedNames has no effect: it
applies only in User mode (ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs:156-161) [V]. So NAM\jdoe and
jdoe@nam.zimmer.com return user not found. Users must type their bare sAMAccountName. Put that on the
login-page notice.
Lockout (AD policy: 5 failed binds per 15 min, per account) [V, LdapAuthService.cs]:
| Failure | Bad binds | Effect |
|---|---|---|
| Wrong user password | 1, on one DC (:164-168, :428-436) |
Final (BadCredentials); no walk to the next DC. Locks only that user, after 5 tries. |
| Wrong service-account password | 2 per login attempt: system-side, so it walks to the next DC (:173-174, :408-415) |
Locks the shared account. |
The shared account is one account for all four apps and both tiers (D1). A mistyped secret in any of the ~7 stores can therefore lock out login everywhere within a few attempts.
ScadaBridge's background group refresh also binds as this account. Its throttle is 5 per 5 min
(ScadaBridge Security/SecurityOptions.cs:132-145) [V], looser than AD's 5 per 15 min. Mitigations:
- D1 asks IT for a no-lockout (or very high threshold) policy on the account.
- §7.1 verifies every store's value before that app restarts.
- §7 does one canary app first.
- §7.6 has a stop rule.
2. Phase A — OtOpcUa code fix (prerequisite for the OtOpcUa cutover)
One OtOpcUa change, one issue, one deploy. It fixes five things.
2.1 Defects [V]
- Options dropped.
LdapOptions.ToLibraryOptions()(src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs:111) copies 12 fields into the library'sLdapOptions(ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs:21).- Dropped:
FallbackServers(0.2.0),BindMode,UserBindTemplateandAcceptDomainQualifiedNames(0.4.0). The config keys are ignored without error, so the AdminUI has no DC failover. - Not config-bound, by design:
ServerCertificateValidationCallback, which is code-only.
- Dropped:
- Zero-role sign-in (review B1).
Security/Endpoints/AuthEndpoints.cs:124-158signs in with an empty role set when the mapper returns no roles, and when the mapper throws: it catches the exception, logs a warning and continues with the baseline roles, which are empty on the real path.- The fallback policy and
AdminUiPolicies.AuthenticatedRead(Security/ServiceCollectionExtensions.cs:192,217) only require an authenticated user, and 18 AdminUI pages useAuthenticatedRead. - No policy requires
Viewer; it is only a default inRoleGrants.razor:126. - With a domain-root
SearchBase, any NAM user with any group would get AdminUI read access, and/auth/tokenwould mint them a token.
- The fallback policy and
- No library logger (review M1).
OtOpcUaLdapAuthService.cs:48buildsLibLdapAuthServicewith no logger, and the library logs only via_logger?.. So the failover log line (sticky preference moved) and the stage diagnostics (failed system-side at …) never appear in OtOpcUa, and §7.6's checks can't be done. - No validation of the new options (review B2). OtOpcUa never runs the library's
LdapOptionsValidator: it constructs the library service directly (OtOpcUaLdapAuthService.cs:48). That answers the old open question.- Calling the library validator would stop OtOpcUa booting on DEV and QA. It requires
ServiceAccountDnin ServiceAccount mode (ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs:81-87), DEV runs on code defaults (Enabled=true,ServiceAccountDn=""), and OtOpcUa validates LDAP on every node (Host/Program.cs:190-195).
- Calling the library validator would stop OtOpcUa booting on DEV and QA. It requires
- Stale docs. The OtOpcUa validator comment (
Host/Configuration/LdapOptionsValidator.cs:12-14) describes a "direct-bind path" that doesn't exist. TheLdapOptionsdoc says the default server is10.100.0.35; the code sayslocalhost.
2.2 The fix
-
Add the four properties with the library's defaults and pass them through
ToLibraryOptions():List<string> FallbackServers = []→IReadOnlyList<string>LdapBindMode BindMode = ServiceAccountstring UserBindTemplate = ""bool AcceptDomainQualifiedNames = true
-
Refuse a zero-role login (the family standard, §3.2). In
/auth/loginand/auth/token:- if the mapped role set is empty, refuse exactly like bad credentials (HTTP 401 / the login page's "invalid or not authorized"), with no cookie and no token;
- if the mapper throws, also refuse (fail closed), log the exception, and do not sign in.
- This reverses the current code comment's intent and the test
Login_when_role_mapper_throws_signs_in_with_no_role_claims(tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/AuthEndpointsIntegrationTests.cs:402). Rewrite that test to assert refusal. - Add tests: empty mapping refused (cookie + token); mapper throws refused (cookie + token); one mapped role signs in.
- DevStub is unaffected: it grants
Administrator.
-
Inject the logger: pass
ILogger<ZB.MOM.WW.Auth.Ldap.LdapAuthService>from DI into theLibLdapAuthServiceconstructor atOtOpcUaLdapAuthService.cs:48. Add a test that a failover writes the library'ssticky preference movedline. -
Validation, OtOpcUa's own validator only. In
Host/Configuration/LdapOptionsValidator.cs, port only:- the library's
FallbackServersrule (each entry parses ashost[:port]); - the
BindMode=User→UserBindTemplatecontains{0}exactly once rule.
Do not port the
ServiceAccountDn-required rule.Add a test that boots the validator with today's DEV-shaped config (
Enabled=true,Server=localhost,ServiceAccountDn="",Transport=Ldaps) and asserts it passes. - the library's
-
Completeness guard. Add a unit test that reflects over the library
LdapOptions' public settable properties.- For each one, set a non-default value on OtOpcUa's options, call
ToLibraryOptions(), and assert the value arrived. CompareIReadOnlyListvalues by content. - Check both directions: every library property is mapped or allow-listed, and no allow-list entry is stale.
- Allow-list:
ServerCertificateValidationCallback. - Mutation check: delete one assignment and the test must fail.
- For each one, set a non-default value on OtOpcUa's options, call
-
Binding test:
Security:Ldap:FallbackServers:0reaches the library options, in JSON and as the env varSecurity__Ldap__FallbackServers__0. -
Docs:
docs/Configuration.mdSecurity:Ldap: the new keys, a NAM two-DC example, the zero-role refusal, and "type the bare sAMAccountName".- Fix the two stale comments from §2.1 item 5.
-
Acceptance A1 (offline): an AdminUI login against the redundant GLAuth pair (
infra/glauth-redundant/,:3893+FallbackServers ["…:3894"]).- Stop the primary: login still succeeds and the log shows
sticky preference moved to …:3894. - A GLAuth user in no mapped group is refused.
- Stop the primary: login still succeeds and the log shows
2.3 Issue text for lmxopcua (not filed by this plan)
Title: AdminUI LDAP: drops
FallbackServers/BindMode/…, signs in users with no mapped role, and the library logs nothingLabels:
type:bug,priority:p1,area:auth,cross-repoBody:
LdapOptions.ToLibraryOptions()(Security/Ldap/LdapOptions.cs:111) dropsFallbackServers,BindMode,UserBindTemplateandAcceptDomainQualifiedNames, so there is no AD DC failover.AuthEndpoints.cs:124-158signs in with an empty role set, including when the mapper throws, andAuthenticatedReadadmits any authenticated user. Once the search base is a whole AD domain, any directory user gets AdminUI read access and a token. The fix refuses empty roles at/auth/loginand/auth/token, and refuses when the mapper throws. This reversesLogin_when_role_mapper_throws_signs_in_with_no_role_claims.OtOpcUaLdapAuthService.cs:48passes no logger to the library, so failover and stage logs are missing. InjectILogger<LdapAuthService>.- Validate the new options in OtOpcUa's own validator: the fallback parse rule and the BindMode/UserBindTemplate rules only. Not
ServiceAccountDn, because DEV boots on code defaults with an empty DN.- A two-way reflection completeness test, a binding test, docs, and fixes to the stale comments.
Acceptance: failover on the redundant GLAuth pair, a zero-role user refused (cookie + token), and a DEV-shaped config boots. Plan:
scadaproj/docs/plans/2026-11-15-nam-ldap-cutover.md§2.
2.4 Ship it
Merge to master and deploy with OtOpcUa's own procedure (docs/deploy/, most recently
2026-11-13-vd03-qa-to-76e73e7d.md): DEV, then QA.
- No LDAP config change goes out with this deploy.
- The zero-role refusal does change behaviour on QA: it is live against the box-local GLAuth today, so any
GLAuth user without a mapped group loses AdminUI read access. Phase 0 checks that the
multi-roleuser maps. - Gate: §7.5 (the OtOpcUa cutover) does not start until the deployed build contains this fix.
2A. Phase A2 — ScadaBridge zero-role refusal (prerequisite for the ScadaBridge cutover)
Defect [V, review B1]:
CentralUI/Auth/AuthEndpoints.cs:59-115signs in (cookie) and issues a bearer at/auth/token(:125) even whenRoleMapperreturns no roles.Dashboard.razor:2andMonitoring/Health.razor:2are bare[Authorize], so any directory user could read them.
Fix:
- In
/auth/loginand/auth/token, an empty mapped role set returns the bad-credentials response, with no cookie and no token. - Apply the same rule to the mid-session group refresh (
DirectoryGroupRefresher). A session whose roles drop to zero is already ended by that refresh (:141-142); keep that behaviour and test it. - Tests: zero roles refused (login + token); one role signs in; a refresh to zero ends the session.
- Docs: the
Component-Security/ CentralUI auth section. - Leave the bare
[Authorize]pages as they are; they are safe once a login requires a role. Add a static test that every page is either policy-gated or relies on that rule.
Issue text for ScadaBridge (not filed):
Title: Central UI /
/auth/tokensign in users whose AD groups map to no roleLabels:
type:bug,priority:p1,area:auth,cross-repoBody:
CentralUI/Auth/AuthEndpoints.cs:59-115signs in and mints a bearer with an empty role set. With a domain-wide NAM search base, any AD user reachesDashboardandMonitoring/Health(bare[Authorize]). The family standard (scadaproj NAM cutover plan §3.2) refuses login when no role is mapped. Fix login and token, keep the refresh's end-on-zero, and add tests.Acceptance: a zero-role user is refused at both endpoints. Plan:
docs/plans/2026-11-15-nam-ldap-cutover.md§2A.
Ship: a ScadaBridge build containing the fix, deployed DEV then QA before §7.4. This is a binary deploy, so it follows the QA pair rules: stop both centrals and start them together.
2B. Phase A3 — HistorianGateway case-insensitive group match (optional hardening)
Defect [review M5]: Dashboard/DashboardOptions.cs:44-45 declares IReadOnlyDictionary<string,string>
with an OrdinalIgnoreCase default. The configuration binder replaces it with a new case-sensitive
Dictionary. The reviewer proved this with an offline binder test; the XML doc says "compared
case-insensitively", which is wrong once it's bound. AD group names are mixed case, so a key whose case differs
from the memberOf CN never matches.
Fix: declare Dictionary<string,string> with StringComparer.OrdinalIgnoreCase, so the binder fills the
existing instance, as mxgw and OtOpcUa do. Add a binder test with a mixed-case key.
Until it ships (and in any case), §4 copies each group name exactly as LdapProbe prints it.
Issue text for historiangw (not filed):
Title:
Dashboard:GroupToRoleis case-sensitive after binding (docs say case-insensitive)Labels:
type:bug,priority:p3,area:authBody: the
IReadOnlyDictionarydefault withOrdinalIgnoreCaseis replaced by the binder's defaultDictionary. Make the property aDictionarywithOrdinalIgnoreCaseand add a binder test with a mixed-case key. Found by the NAM cutover review (scadaproj plan §2B).
3. The family standards (DECIDED 2026-11-15)
3.1 One AD group grants exactly one role per app
A user's roles in an app are the union of the roles of every mapped group they belong to. No app maps one group
to two roles. Each deployment maps only its own _DEV/_QA/_PROD suffix.
Why:
-
Every schema already enforces or implies it, so no code change is needed:
App Mapping shape [V] OtOpcUa Dictionary<string,string> GroupToRole(Security/Ldap/LdapOptions.cs:84), unioned with the DB overlayLdapGroupRoleMapping(OtOpcUaGroupRoleMapper.cs). The overlay's only unique index is(LdapGroup, ClusterId), filtered, so two system-wide rows per group are possible; the rule forbids them and Phase 0 checks.ScadaBridge table LdapGroupMappings, withHasIndex(LdapGroupName).IsUnique()(ConfigurationDatabase/Configurations/SecurityConfiguration.cs:26).RoleMapperunions roles across different groups (Security/RoleMapper.cs:48-60).MxAccessGateway Dictionary<string,string>(Configuration/DashboardOptions.cs:69). The validator allows onlyAdministrator/Viewer(GatewayOptionsValidator.cs:506-517).HistorianGateway IReadOnlyDictionary<string,string>(Dashboard/DashboardOptions.cs:44). Case-sensitive in practice (§2B). -
Grants stay readable in AD, where IT owns membership.
-
It removes a silent failure: the old two-row ScadaBridge insert hits the unique index.
Rejected: several roles per group. It would need a ScadaBridge migration and matching changes in the other apps. It stays a future option.
3.2 No role, no login (added by review B1)
Every app refuses sign-in, both cookie and token, when the user's groups map to no role. That includes the case where the role lookup fails.
| App | Before the fixes [V] | Change (status 2026-11-17: merged, not deployed) |
|---|---|---|
| MxAccessGateway | refuses (Dashboard/DashboardAuthenticator.cs:66-70) |
none |
| HistorianGateway | refused zero roles; a throwing mapper escaped as a 500 | historiangw#136, f84af29: case-insensitive GroupToRole, a mapper fault is refused like a bad password, both logged at Warning (Dashboard/DashboardAuthenticator.cs, the MapAsync catch and the roles.Count == 0 branch) |
| OtOpcUa | signed in with no roles, even when the mapper threw | §2 (Phase A): lmxopcua#913, cdf433a35 |
| ScadaBridge | signed in with no roles: cookie, /auth/token, management HTTP Basic/Bearer |
§2A (Phase A2): #144 d48e1667 (cookie, token, session refresh) + #145 2e0ce81e (management/CLI, audit, debug stream) |
Consequence for the matrix: read access now requires a mapped group. In OtOpcUa, Viewer is not checked
by any policy (pages use AuthenticatedRead), so a Viewer cell means "holds a role, so may sign in and read".
That is why APP_IDE_RO_<T> → Viewer stays, and why HIST_ADMIN_<T> → Viewer in OtOpcUa is meaningful.
3.3 Consequence for ScadaBridge Deployer
APP_IDE → Designer only. In ScadaBridge, Administrator does not imply Deployer: each policy requires its
own role (Security/AuthorizationPolicies.cs:190-202). No existing group means "deploy", so:
- a new AD group
UG_WW_APP_DEPLOY_{DEV,QA}(D6) maps toDeployer; - until it exists on a tier, the ScadaBridge cutover does not run there, because nobody could deploy.
The other apps have no Deployer role; OtOpcUa deploy is ConfigEditor = Administrator/Designer,
AdminUiPolicies.cs:18.
4. The tier mappings (generated from ldap_mapping.md §4.1)
These are platform groups only. Plant-floor, site and batch groups are not in this cutover (D5), and CITRIX_*
is never mapped. Copy each group name exactly as LdapProbe prints it (§2B; D7). The names below are as
observed on 2026-08-12.
Group (<T> = DEV or QA) |
OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
UG_WW_APP_ADMIN_<T> |
Administrator | Administrator | Administrator | Administrator |
UG_WW_APP_IDE_<T> |
Designer | Designer | Viewer | — |
UG_WW_APP_DEPLOY_<T> (new, D6; only once it exists) |
— | Deployer | — | — |
UG_WW_APP_IDE_RO_<T> |
Viewer | Viewer | Viewer | Viewer |
UG_WW_HIST_ADMIN_<T> |
Viewer | — | — | Administrator |
UG_WW_BTMGR_RO_QA (QA only; empty as of 08-12) |
— | Viewer | — | Viewer |
4.1 HistorianGateway (DEV: appsettings.Production.json; QA: NSSM env)
"Ldap": { /* §6 */ },
"Dashboard": {
"DisableLogin": false,
"GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_HIST_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
}
On QA, remove every Dashboard__GroupToRole__SCADA-* entry first, then set:
Dashboard__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_HIST_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer
Dashboard__GroupToRole__UG_WW_BTMGR_RO_QA=Viewer
4.2 MxAccessGateway (the overlay file for the box's environment name; Phase 0 records it)
The base appsettings.json ships GwAdmin/GwReader mappings (:86-89) [V], and dictionaries merge
across overlays. So the overlay alone can't remove them, and they can't be neutralised either, because the
validator accepts only Administrator/Viewer as a value. Delete the two entries from the box's base
appsettings.json (backed up). A redeploy restores the base file, so the mxgw deploy notes must carry the
deletion forward.
Then set:
"MxGateway": {
"Ldap": { /* §6 */ },
"Dashboard": {
"DisableLogin": false,
"GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_DEV": "Viewer", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
}
}
On QA, use the _QA suffix and delete GwAdmin/GwReader the same way. Acceptance A4 reads the effective
config (§8), not just the edited file.
4.3 OtOpcUa (service env, both tiers)
Security__Ldap__GroupToRole__UG_WW_APP_ADMIN_<T>=Administrator
Security__Ldap__GroupToRole__UG_WW_APP_IDE_<T>=Designer
Security__Ldap__GroupToRole__UG_WW_APP_IDE_RO_<T>=Viewer
Security__Ldap__GroupToRole__UG_WW_HIST_ADMIN_<T>=Viewer
- DB overlay: the RoleGrants page /
LdapGroupRoleMappingmust hold no rows for these groups, or only the same role. DriverOperatoralso honours the roleOperator(ServiceCollectionExtensions.cs:201-202); nothing maps to it here.
4.4 ScadaBridge (central database)
Preferred: the management surface, so the change is audited by the ManagementActor and the 30 s cross-central role-mapping cache is invalidated.
- Use the CLI
securityrole-mapping commands:scadabridge security --helplists them. Run them on the target tier with an explicit--url, while DEV login is still off (the dev user holds every role). - On QA, run them as the current GLAuth admin before the cutover.
- The Central UI's role-mapping page is equivalent.
Add one mapping per group:
UG_WW_APP_ADMIN_<T>→AdministratorUG_WW_APP_IDE_<T>→DesignerUG_WW_APP_IDE_RO_<T>→ViewerUG_WW_APP_DEPLOY_<T>→Deployer, only if that group exists (D6)- QA only:
UG_WW_BTMGR_RO_QA→Viewer
Fallback: direct SQL (idempotent; refuses to change a conflicting row). It bypasses the audit and the cache invalidation, so record it in the deployment record and wait 30 s (or restart Central) before testing.
SET XACT_ABORT ON;
BEGIN TRAN;
DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL);
INSERT @t VALUES (N'UG_WW_APP_ADMIN_DEV', N'Administrator'), (N'UG_WW_APP_IDE_DEV', N'Designer'), (N'UG_WW_APP_IDE_RO_DEV', N'Viewer');
-- Only if D6's group exists: INSERT @t VALUES (N'UG_WW_APP_DEPLOY_DEV', N'Deployer');
-- QA: _QA suffix, plus (N'UG_WW_BTMGR_RO_QA', N'Viewer').
IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role)
THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1;
INSERT dbo.LdapGroupMappings (LdapGroupName, Role)
SELECT t.LdapGroupName, t.Role FROM @t t
WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName);
COMMIT;
SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id;
Names [V]:
- Table
LdapGroupMappings. - Columns
LdapGroupNamenvarchar(500)andRolenvarchar(100)(SecurityConfiguration.cs:16-24). - Role values are the
Security/Roles.cs:45-56constants.
The SCADA-* seed rows (Ids 1–5) name GLAuth groups that don't exist in NAM, so they grant nothing; leave them.
5. Decisions (owner or IT): settle before the cutover (§7)
| # | Decision | Options | Recommendation |
|---|---|---|---|
| D1 | Service account (owner request: one dedicated LDAP-lookup account shared by the four services, scadaproj#58) | Target: the dedicated account from scadaproj#58, shared across all four apps; one account for DEV and QA unless IT prefers one per tier. Interim (DEV only): svc_igntadmin@nam.zimmer.com, only with the Ignition team's agreement. It is their admin account, and their rotations or lockouts would hit us. |
Target from scadaproj#58, a QA prerequisite. Add to the scadaproj#58 ask: a no-lockout policy (or a very high threshold) on the account, because it is shared by four apps and two tiers and a wrong secret costs 2 bad binds per login (§1.3). BindMode=ServiceAccount everywhere. Rotation = re-seed nam-ldap-svc-password in every store and restart each app, since values resolve only at startup (§7.1). |
| D2 | Turn login on where DEV is deliberately open (HG; mxgw = mxaccessgw#231 item 3; ScadaBridge) | Turn it on / leave DEV open | Turn it on. DEV is where problems should surface. |
| D3 | Automation identity (CLI, deploy and drill scripts). QA: today GLAuth multi-role. DEV: today the login-off dev user (ManagementAuthenticator.cs:63-71), which stops working the moment DEV ScadaBridge login turns on. The same applies to OtOpcUa scripts if DEV Security__Auth__DisableLogin=true (Phase 0). |
(a) A NAM user-type identity per tier in APP_ADMIN_<T> + APP_IDE_<T> + APP_DEPLOY_<T>, password in the Mac Keychain. The scadaproj#58 account is lookup-only and should not double as a login unless IT agrees. (b) An interim DEV identity (a named admin's account) for DEV only. |
(a). A precondition of the DEV ScadaBridge step (or (b) as a DEV interim) and of the QA ScadaBridge step. Also of the OtOpcUa step if Phase 0 finds DEV OtOpcUa login off. |
| D4 | TLS trust | (a) Install zimmer-rootca in LocalMachine\Root, AllowInsecure=false. (b) AllowInsecure=true: accepts any certificate (ZB.MOM.WW.Auth.Ldap/Internal/NovellLdapConnection.cs:41-46), so an active man-in-the-middle can read the service-account and every user's password. (c) Pin: in OtOpcUa, HG and ScadaBridge this is code-only (ServerCertificateValidationCallback), but mxgw has a config-bound strict pin, MxGateway:Ldap:ServerCertificateThumbprintSha256 (LdapServerCertificatePin.cs:9-21). |
(a). (b) is DEV only and short-lived, if at all. For mxgw: remove any GLAuth pin left in config (§7.3), and optionally pin the NAM DC leaf thumbprints. The certs expire 2027-03, so a pin must be renewed. |
| D5 | Plant-floor and site groups | now / later | Later (ldap_mapping.md §6). |
| D6 | New group UG_WW_APP_DEPLOY_{DEV,QA} (ScadaBridge Deployer) |
ticket it / the rejected multi-role code (§3.1) | Ticket it. Blocks the ScadaBridge step on each tier. |
| D7 | Group membership and exact names | — | Per person: dotnet run --project tools/LdapProbe -- --insecure --svc '<svc account UPN>' -u <sAMAccountName> (service-account path; prompts for the service password). Or, as the person, --insecure --self-bind -u <sAMAccountName> (binds as them; --svc is ignored). Families: -g 'UG_WW_*'. Missing people get a ServiceNow membership request, not a wider mapping. Copy group names exactly (§2B). |
6. The LDAP block (each app's section)
The sections are: HG Ldap, mxgw MxGateway:Ldap, ScadaBridge ScadaBridge:Security:Ldap
(Security/ServiceCollectionExtensions.cs:24), OtOpcUa Security:Ldap.
{
"Enabled": true,
"Server": "uswap-vdc-nam1.nam.zimmer.com",
"Port": 636,
"Transport": "Ldaps",
"AllowInsecure": false, // D4
"FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ], // OtOpcUa: only with the §2 build
"SearchBase": "DC=nam,DC=zimmer,DC=com",
"ServiceAccountDn": "<svc account UPN from scadaproj#58>", // D1; DEV interim: svc_igntadmin@nam.zimmer.com (Ignition agreement)
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
"UserNameAttribute": "sAMAccountName",
"DisplayNameAttribute": "name",
"GroupAttribute": "memberOf",
"BindMode": "ServiceAccount"
}
mxgw only: no ServerCertificateThumbprintSha256 unless D4 chooses to pin the NAM DCs.
The env form follows the usual pattern, e.g. Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636.
7. Runbook
Order:
- DEV: HistorianGateway (the canary, §7.6 rule 6) → MxAccessGateway → ScadaBridge (after §2A, D3, D6) → OtOpcUa (after §2).
- A soak of a few working days with real logins.
- QA in the same order. QA also needs scadaproj#58 (QA never runs on
svc_igntadmin). If DEV used the interim account, switch DEV to the scadaproj#58 account during the soak, so that QA repeats a configuration already proven on DEV.
One app at a time: edit, restart, verify, then the next.
Remote execution only within the winrm deploy scope: preflight, backup, edit, restart, verify, rollback.
Files go over SMB and SQL runs from the Mac. servecli is retired.
Restart rules (review M4). This is a config-only change: the same binary, no migration, no wire change. So restarts are rolling, standby first:
| Pair | How | Why |
|---|---|---|
| OtOpcUa QA pair | Restart the standby (ServiceLevel 240/90), confirm it is healthy and back at its level, then the primary. The primary's restart is a failover; ScadaBridge's OPC UA client follows ServiceLevel, as WarmCanary showed. | The stop-both/start-both rule in OtOpcUa's records is for binary/schema upgrades. Config-only allows rolling [I: confirm against OtOpcUa's deploy notes in Phase 0]. A bad LDAP key stops a node booting, which is why the standby goes first. |
| ScadaBridge QA centrals | Standby central first, confirm it rejoins (one active, /health/active), then the active one, which gives a takeover of about 8–11 s. Sites are not restarted. |
Central LDAP config only. The LocalDb stop-together rule applies to the site pairs' sync, which this change doesn't touch. The stop-together rule for binaries (§2A) does not apply. |
| Single nodes (DEV; HG/mxgw on each QA box) | One restart each. | — |
A pair restarted together is an outage of its data plane (OPC UA or the site data path). Don't do it for this change.
7.0 Phase 0 — preflight (read-only; deploy scope; per box vd03, vq03, vq04)
- DC reachability:
Test-NetConnectionto nam1 and nam2 on 636 and 389. - TLS: check the chain to nam1:636 (
SslStream, orcertutil -verify) and whetherzimmer-rootcais inLocalMachine\Root. This settles D4. - Service-account bind, once from the Mac:
LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D '<svc account UPN>' -W -b "DC=nam,DC=zimmer,DC=com" -s base dn - Current config per app, kept for rollback:
- hash the appsettings;
- export the NSSM/service env;
- list every
Ldap__*,*__Ldap__*,*DisableLogin*,*__GroupToRole__*entry; - record OtOpcUa's
Security__Auth__DisableLoginon all three boxes; - record mxgw's environment name (it decides the overlay file; vd03 can't be Production while
DisableLogin=true); - record mxgw's
ServerCertificateThumbprintSha256(a leftover GLAuth pin blocks NAM); - record mxgw's base
GwAdmin/GwReaderentries.
- QA mxgw
dc=zbvsdc=scadabridge: does a test login work today? This decides the rollback target. - Secrets per app:
- the store path, master-key source and KEK location for the §7.1 fallback. The family default is
MasterKey:Source=Environment/ZB_SECRETS_MASTER_KEY; file-based KEKs exist on some boxes. Record which each app uses; don't assume. - ScadaBridge Central
Secrets:Store:Provider. QA is the shared SQL store (2026-11-13 QA pack §1); DEV is [I]Sqlite. - OtOpcUa
Secrets:Replication:Enabled(Host/Configuration/SecretsRegistration.cs:21). If it's on, the QA pair shares the secret through Akka replication, and "one store per box" becomes "seed once". - HG KEK health on vd03 (historiangw#134).
- the store path, master-key source and KEK location for the §7.1 fallback. The family default is
- OtOpcUa DB overlay: the
LdapGroupRoleMappingrows (§4.3). - Builds:
- the deployed OtOpcUa build contains §2 (required before §7.5);
- the deployed ScadaBridge build contains §2A (before §7.4);
- on QA, the GLAuth
multi-roleuser still maps to a role after §2/§2A (§2.4).
7.1 Seed and verify the secret (before any config change)
- Name:
nam-ldap-svc-password, the same in every store.- Values resolve only at startup, so a rotation is re-seed every store plus restart every app (§7 restart rules).
- An unresolved
${secret:}throwsSecretNotFoundExceptionat boot (SecretReferenceExpander.cs:74-77), so the app won't start.
- Preferred: each app's
/admin/secretspage. It writes through the app's own provider, which is correct for ScadaBridge's shared SQL store and for a replicated OtOpcUa store.- DEV HG, mxgw and ScadaBridge are still login-off at this point. On QA, use the current GLAuth admin.
- Fallback: the
secretCLI, with the store and master-key settings from Phase 0 step 6, per app:$env:Secrets__SqlitePath = "<that app's store, Phase 0>" $env:Secrets__MasterKey__Source = "<Environment|File, Phase 0>" # family default: Environment (ZB_SECRETS_MASTER_KEY) $env:Secrets__MasterKey__FilePath = "<kek, only if File>" secret.exe set nam-ldap-svc-password # prompts; never pass the value as an argument - Stores:
- HG and mxgw: per box.
- OtOpcUa: per box, or once if replication is on.
- ScadaBridge: once per tier through Central. The site does no LDAP.
- Verify the VALUE before the app's restart (review M2): read it back (
get) and bind with it once using LdapProbe (--svc '<svc account UPN>', pasting the read-back value). One good bind proves the stored secret. A wrong secret found here costs 1–2 bad binds, instead of 2 per user login after the restart.
7.2 HistorianGateway (DEV: file; QA: NSSM env). The canary.
- Back up the file or env.
- Write §6 and §4.1. On DEV, set
DisableLogintofalse. On QA, replace theLdap__*entries and remove theSCADA-*GroupToRoleentries. - Restart
ZbHistorianGateway. Run §7.6 fully, including the failover check, before touching any other app.
7.3 MxAccessGateway (the file for its environment name)
- Back up the base and overlay files.
- Write
MxGateway:Ldap(§6) andGroupToRole(§4.2), and delete the baseGwAdmin/GwReader. - Remove the
ServerCertificateThumbprintSha256pin, or replace it with the NAM DC leaf thumbprints (D4). - On DEV, set
DisableLogintofalse(D2, mxaccessgw#231 item 3). If that makes the box eligible to run as Production, record the environment name. Don't change it in this step. - Leave the API-key store under
data\and the pepper env alone. - Restart, then run §7.6.
7.4 ScadaBridge (Central only)
Preconditions:
- §2A deployed;
- D6:
UG_WW_APP_DEPLOY_<T>exists, has members, and is mapped; - D3: the automation identity works. On DEV it is needed before login turns on, because the dev-user path ends then.
- Back up
appsettings.Central.jsonand the env. - Add the role mappings (§4.4, CLI preferred).
- Write
ScadaBridge:Security:Ldap(§6). - On DEV, remove BOTH
ScadaBridge:Security:Auth:DisableLoginandAllowOutsideDevelopment. Removing only the second whileDisableLogin=truemakesDisableLoginGuardcrash-loop the service. - On QA, replace the GLAuth values.
- Restart: DEV, the single central. QA, rolling, standby central first (§7 restart rules).
- Re-point every script that used
multi-role(or relied on login-off) at the D3 identity:apply.sh, the drill and deploy packs. Confirm/auth/tokenworks with it.
7.5 OtOpcUa AdminUI (service env; only after §2 is deployed)
-
Back up the env.
-
Add §6 as
Security__Ldap__*and the §4.3GroupToRole. Keep the files'Transport/DevStubMode. -
If
Security__Auth__DisableLogin=true, set it tofalse, and if scripts relied on it, have D3 in place first. -
Restart: DEV, the single node. QA, rolling, standby first (§7 restart rules). Watch ServiceLevel and the ScadaBridge
WarmCanaryconnection across the primary's restart. -
Existing sessions: AdminUI cookies and bearer tokens issued under GLAuth stay valid after the switch until they expire. Either:
- invalidate them (restart both nodes, if the cookie and data-protection keys are per process [I], or rotate the signing or data-protection key); or
- record their remaining lifetime and accept it.
ScadaBridge needs no such step: its group refresh ends sessions whose roles change (
DirectoryGroupRefresher.cs:141-142).
7.6 Verify (each app, right after its restart)
- Negative control first: a bogus user and password. Expect a clean "invalid / not authorized" and no
failed system-sidewarning.- A
service-bind-stage warning means the stored service secret is wrong. Stop: every further attempt costs the shared account 2 bad binds. - A
connect-stage warning means network or TLS. - (An unresolved
${secret:}never gets this far; the app fails at boot withSecretNotFoundException.) - (OtOpcUa logs these lines only with the §2 logger fix.)
- A
- One real login per mapped role: Administrator, Designer, Viewer, and Deployer for ScadaBridge. The UI shows exactly that role, and the user typed a bare sAMAccountName (§1.3).
- A user whose groups map to no role is refused, on the login page and at the token endpoint where one
exists (OtOpcUa and ScadaBridge
/auth/token). This is the §3.2 standard: in OtOpcUa and ScadaBridge it holds only with the §2/§2A builds; mxgw and HG already refuse. ${secret:}proof: no*ServiceAccountPasswordsits in env, and the stored value was bind-verified (§7.1).- Failover:
- On HG (the canary), once per tier: block nam1:636 briefly. Login still works and the log shows
sticky preference moved; then restore. - OtOpcUa gets its own nam1→nam2 check (acceptance A2).
- On HG (the canary), once per tier: block nam1:636 briefly. Login still works and the log shows
- Stop rule (review M2): stop the cutover on 2 failed logins attributable to the service account, across
all apps, within 15 minutes. Count
service-bindwarnings, not user typos. Diagnose before any retry; ask IT to check the account's lockout state.
7.7 Rollback (per app)
- Restore the backed-up files or env and restart. On QA, use the same rolling order: standby first.
- mxgw: also restore its base
appsettings.json(theGwAdmin/GwReaderentries) and any pin.
- mxgw: also restore its base
- Point the scripts back at
multi-role(QA), or back at login-off (DEV ScadaBridge). - If the shared account got locked, IT clears the lockout. Until then, every NAM-backed app is down for login; roll those apps back to GLAuth or login-off first.
- The seeded secrets are inert once nothing references them. The ScadaBridge
UG_WW_*mapping rows grant nothing while login is off or on GLAuth. - Keep the §2/§2A/§2B builds. The zero-role refusal is the family standard and stands on its own.
8. Acceptance
- A1: the §2 offline tests pass:
- failover on the redundant GLAuth pair with the library's log line;
- zero-role refusal (cookie + token, including the mapper-throws case);
- a DEV-shaped config boots;
- the reflection guard fails under its mutation.
- The §2A tests pass too.
- A2: on each tier, the OtOpcUa AdminUI login survives nam1 being blocked, through nam2.
- A3: every §7.6 check passes for each tier and app.
- A4: no app maps any group to two roles, and no stale GLAuth mapping remains. Check the effective
config: env plus all overlays plus base, including mxgw's
GwAdmin/GwReader, and the ScadaBridge table. - A5: automation runs on the D3 identity. Nothing uses
multi-role, or relies on login-off, on a cut-over app. - A6: QA (and DEV after the soak) binds as the scadaproj#58 account and never as
svc_igntadmin.
9. Open questions
Does OtOpcUa run the library's validator?Answered: no (§2.1 item 4). The validation is ported selectively (§2.2 item 4).- Does QA mxgw login work today (
dc=zbvsdc=scadabridge)? Phase 0 step 5. - What is the HG KEK state on vd03? (historiangw#134)
- Is OtOpcUa's AdminUI cookie or data-protection key per process or shared? That decides the §7.5 step 5 method [I].
- Does OtOpcUa's deploy procedure allow a rolling restart for config-only changes? Confirm against its records (§7 restart rules).
- Do the
UG_WW_*memberships and exact names still match 2026-08-12? D7 re-enumerates. - ignitionoee: still on hold.
10. Corrections from the 2026-08-13 plan
| 08-13 plan said | Fact (2026-11-15) |
|---|---|
| Phase 1: redeploy to Auth 0.2.1 | Every build already runs 0.4.1 (ScadaBridge 0.5.0). Phase deleted. |
Preflight over servecli :2222 |
servecli is retired; the winrm deploy scope plus SMB plus SQL replace it. |
| mxgw on vd03 has LDAP login on | DisableLogin=true (and so it isn't running as Production). |
| One target block for all four apps | OtOpcUa drops FallbackServers and three more options, and logs nothing. Fixed in §2. |
| A zero-group user is denied in every app | Only mxgw and HG refuse. OtOpcUa and ScadaBridge sign in with no roles. Fixed in §2/§2A; the standard is §3.2. |
ScadaBridge INSERT INTO LdapGroupMapping, two rows for APP_IDE |
The table is LdapGroupMappings, unique LdapGroupName. One-group-one-role plus the new D6 group; CLI preferred over SQL. |
ldap_mapping.md: ScadaBridge can grant two roles per group |
It can't. Rewritten as the canonical matrix. |
| Seed the ScadaBridge central and site stores with the CLI | Seed once through Central /admin/secrets; the site does no LDAP. |
| Stop after 2 failures; every failed login costs both DCs | A wrong user password costs 1 bind. A wrong service secret costs 2 per login and locks the shared account. New stop rule, a no-lockout ask and value verification. |
Service account svc_igntadmin now, dedicated "later" |
Target: scadaproj#58, a QA prerequisite; svc_igntadmin is a DEV interim only, with Ignition's agreement. |
| OtOpcUa before ScadaBridge | OtOpcUa last, after its fix. |
| (silent) automation | DEV and QA automation lose their identity on cutover. D3 is a precondition. |
| (silent) restarts | Rolling, standby first, for config-only changes. Restarting a pair together is a data-plane outage. |
11. Review 2026-11-15: findings and disposition
| # | Finding | Disposition |
|---|---|---|
| B1 | OtOpcUa and ScadaBridge sign in users with zero mapped roles (OtOpcUa even when the mapper throws); AuthenticatedRead/bare [Authorize] then grant read access to any AD user |
Applied. New standard §3.2; OtOpcUa fix §2.2 item 2 (refuses on throw too, reversing Login_when_role_mapper_throws_signs_in_with_no_role_claims); ScadaBridge Phase A2 §2A with issue text and a deploy; Viewer semantics §3.2; §7.6 step 3 and ldap_mapping.md §1 rewritten. |
| B2 | Calling the library validator would stop OtOpcUa booting on DEV/QA (ServiceAccountDn required) |
Applied. §2.2 item 4 ports only the FallbackServers and BindMode/UserBindTemplate rules, with a DEV-shaped boot test. Open question 1 answered (§2.1 item 4). |
| M1 | No logger passed to the library service | Applied. §2.2 item 3. |
| M2 | Lockout on the shared account; §1.3 had it backwards | Applied. §1.3 table; D1 no-lockout ask; §7.1 value verification; HG as canary; §7.6 rule 6 stop rule. |
| M3 | DEV automation breaks when DEV login turns on | Applied. D3 covers DEV (dev-user path, ManagementAuthenticator.cs:63-71); a precondition in §7.4/§7.5. |
| M4 | "Data plane unaffected" was misleading; pairs restarted together are an outage | Applied. §7 restart rules: rolling, standby first, for config-only changes. The OtOpcUa rolling allowance is [I] pending Phase 0 / open question 5. |
| M5 | HG group match is case-sensitive after binding | Applied. §2B (optional Phase A3) with issue text; exact names from LdapProbe in §4 and D7. |
| M6 | mxgw config-bound certificate pin | Applied. Phase 0 step 4; §7.3 step 3; D4 corrected. |
| m | §2.4 / §7.0 cross-references | Applied (→ §7.5). |
| m | Unresolved ${secret:} fails at boot |
Applied (§7.1, §7.6). |
| m | Rotation needs restarts | Applied (D1, §7.1). |
| m | AcceptDomainQualifiedNames only applies in User mode |
Applied (§1.3; users type the bare sAMAccountName). |
| m | D7 command was wrong | Applied (-u with --svc, or --self-bind -u). |
| m | Secrets CLI hardcoded File |
Applied (Phase 0 step 6 supplies the values; default noted). |
| m | OtOpcUa secret replication | Applied (Phase 0 step 6; §7.1 stores). |
| m | mxgw vd03 can't be Production with DisableLogin |
Applied (§1.2; Phase 0 step 4; §7.3 step 4). |
| m | mxgw base GwAdmin/GwReader merge |
Applied (§4.2 delete from base; A4 checks the effective config). |
| m | Existing OtOpcUa sessions survive cutover | Applied (§7.5 step 5; ScadaBridge self-ends via refresh). |
| m | Rollback gaps | Applied (§7.7: scripts, AD lockout, rolling on QA). |
| m | AllowInsecure accepts any certificate |
Applied (D4(b)). |
| m | Stale OtOpcUa validator comment | Applied (§2.1 item 5, §2.2 item 7). |
| m | ScadaBridge SQL bypasses audit and cache | Applied (§4.4: CLI/UI preferred; SQL fallback with a 30 s wait). |
| n | LdapOptions doc default 10.100.0.35 vs code localhost |
Applied (§2.1 item 5). |
| n | Reflection test: compare lists by content, both directions | Applied (§2.2 item 5). |
| n | §4.4 DEPLOY row conditional | Applied. |