Files
scadaproj/docs/plans/2026-11-15-nam-ldap-cutover.md
T

66 KiB
Raw Blame History

NAM AD LDAP cutover — DEV vd03, then QA vq03/vq04 (DEV + QA EXECUTED 2026-11-19 on the interim account)

Status lives in Gitea. Execution is tracked by scadaproj#6. The dedicated, shared LDAP-lookup service account is scadaproj#58, a QA prerequisite (§5, D1). This file is the design and runbook. It supersedes 2026-08-13-vd03-nam-ldap-cutover.md (§10 lists what changed). The independent review of 2026-11-15 is applied (§11).

Written 2026-11-15. Status: PLAN ONLY. Nothing here has been executed. EXECUTED ON DEV 2026-11-19 (§0): all four apps on vd03 log in against NAM AD. Record: docs/deploy/2026-11-19-vd03-nam-ldap/. The DC-failover checks (§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's RequireHttpsCookie on a plain-HTTP AdminUI (record, finding 1). QA plan: §0Q (written 2026-11-19, not executed; QA hold stands). Updated 2026-11-19: §0 is the DEV vd03 execution plan using the groups and accounts that exist today, so it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full prerequisites in §5–§7, and the QA deploy hold still stands.


0. DEV vd03 now, with today's groups and accounts (2026-11-19)

0.1 What the directory holds today

Re-enumerated 2026-11-19 from the Mac (ldapsearch over LDAPS to nam1, bound as svc_igntadmin; that bind is the D1 interim's single verification bind).

  • Groups: 41 UG_WW_*, the 39 from 08-12 plus UG_WW_PAR_CVD_Oper and UG_WW_PAR_CVD_Super. Neither is a platform group, so neither is mapped. There is still no UG_WW_APP_DEPLOY_* group.
  • Names: the DEV platform names are exactly as in §4. Copy them with this case.
DEV group Members (2026-11-19)
UG_WW_APP_ADMIN_DEV (7) chamalas, millerh5, benjamd2, marting4, dohertj2, oswaltk, Luis Bastidas
UG_WW_APP_IDE_DEV (6) chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas
UG_WW_APP_IDE_RO_DEV (2) benjamd2, oswaltk
UG_WW_HIST_ADMIN_DEV (7) the same 7 people as APP_ADMIN_DEV
UG_WW_CITRIX_DEV (6) never mapped; culverj holds only this group, so they get no role on DEV

Accounts used on DEV (no new accounts):

Use Account Groups and resulting roles Credential
LDAP lookup (D1 interim) svc_igntadmin@nam.zimmer.com No UG_WW_* group, so it cannot sign in to any app, which is correct for a lookup account Mac Keychain, service btdb-manager/ldap. Seeded into each store as nam-ldap-svc-password
Positive login and automation (D3 option b) dohertj2 APP_ADMIN_DEV + APP_IDE_DEV + HIST_ADMIN_DEV. Gets Administrator everywhere, plus Designer in OtOpcUa and ScadaBridge, plus Deployer in ScadaBridge (§0.2) Mac Keychain (NAM\dohertj2)
Negative control: zero roles adm_dohertj2 3 groups, none UG_WW_*. The library passes it (memberOf is non-empty), the app maps no role, and §3.2 must refuse it Mac Keychain (NAM\adm_dohertj2)
Negative control: bad credentials a bogus user/password — —

Viewer is not exercised live on DEV. Only benjamd2 and oswaltk hold APP_IDE_RO_DEV, and both also hold APP_ADMIN_DEV. The Viewer mapping is checked by reading the effective config back (A4). A real Viewer-only login waits for QA or for a volunteer.

0.2 Deviations from the full plan (DEV only)

Full plan DEV now Why it's acceptable Ends when
D1: dedicated account from scadaproj#58 svc_igntadmin The interim §5 already allows on DEV. Tell the Ignition team before §0.4 step 3: a wrong secret here locks their admin account (2 bad binds per login, §1.3) scadaproj#58 is delivered; swap it in during the soak (§7)
D3: a NAM automation account dohertj2 (option b) It holds every DEV role the scripts need The D3 account exists
D6: UG_WW_APP_DEPLOY_DEV → Deployer UG_WW_HIST_ADMIN_DEV → Deployer, in ScadaBridge only Still one group, one role per app (§3.1): ScadaBridge maps HIST_ADMIN to nothing else. Its members are identical to APP_ADMIN_DEV, so the effect is "DEV admins can deploy", the SoD collapse already accepted for this directory (ldap_setup.md). It is a DEV-only row D6's group exists: add APP_DEPLOY_DEV → Deployer, then delete the HIST_ADMIN_DEV row
D4: zimmer-rootca in LocalMachine\Root Phase 0 checks it. vd03 is domain-joined, so the root is expected to be there via GPO [I]. If it isn't, run DEV with AllowInsecure=true and record it DEV only, short-lived (D4 b) The root is installed
D7: membership confirmed per person Done for DEV by §0.1 — —

Unchanged: the standards (§3), the LDAP block (§6, with ServiceAccountDn: svc_igntadmin@nam.zimmer.com), the restart rules, the §7.6 verification and stop rule, and the rollback (§7.7).

0.3 Builds on vd03

App On vd03 (2026-11-19) Carries Action
HistorianGateway f84af29 #136 (case-insensitive GroupToRole, mapper fault refused) none
MxAccessGateway e74918cc already refuses zero roles none
OtOpcUa cdf433a35 #913 (§2): options passthrough, zero-role refusal, library logger none
ScadaBridge 061432c8 neither #144 nor #145 Deploy main 2e0ce81e (§0.4 step 1)

ScadaBridge 061432c8..2e0ce81e is 11 commits: #144, #145 and the #116 leader-auto-down default (already drilled on vd03, 2026-11-15). No migration and no wire change. Use the procedure in docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md (single central + site; take a DB backup first). With login still off, the dev user holds every role, so #144/#145 have no visible effect until step 6.

0.4 Order of work on vd03

All remote steps stay inside the winrm deploy scope (§7). One app at a time; each one passes §7.6 before the next.

  1. ScadaBridge binary deploy 2e0ce81e. Verify with the deploy's normal checks: health, the site connected, the CLI works as the login-off dev user.
  2. Phase 0 on vd03 (§7.0), reduced: DC reach on 636 from vd03; the zimmer-rootca check (decides D4); back up every app's config and env; record OtOpcUa Security__Auth__DisableLogin (expected true), mxgw's environment name, its ServerCertificateThumbprintSha256 and base GwAdmin/GwReader; each app's secrets store and KEK source; the OtOpcUa LdapGroupRoleMapping rows. Builds per §0.3. The service-account bind was done 2026-11-19 (§0.1), so don't repeat it.
  3. Seed nam-ldap-svc-password in each store (§7.1): HG, mxgw and ScadaBridge Central through their /admin/secrets pages (login is still off on all three), and OtOpcUa's store with the secret CLI or its page. Verify each read-back value with one LdapProbe bind as svc_igntadmin.
  4. HistorianGateway (the canary), §7.2. §6 + §4.1 DEV mapping, Dashboard:DisableLogin=false. Run §7.6 fully, including the nam1→nam2 failover check.
  5. MxAccessGateway, §7.3. §6 + §4.2 DEV mapping; delete the base GwAdmin/GwReader; remove any GLAuth pin; DisableLogin=false.
  6. ScadaBridge, §7.4.
    • a. While login is still off, add the role mappings with the CLI (--url the vd03 central): UG_WW_APP_ADMIN_DEV→Administrator, UG_WW_APP_IDE_DEV→Designer, UG_WW_APP_IDE_RO_DEV→Viewer, UG_WW_HIST_ADMIN_DEV→Deployer (DEV interim, §0.2). The §4.4 SQL fallback works too: put the HIST_ADMIN_DEV row in its @t.
    • b. Write ScadaBridge:Security:Ldap (§6). Remove both DisableLogin and AllowOutsideDevelopment.
    • c. Restart the central. Re-point scripts (apply.sh, drill and deploy packs) at dohertj2, and confirm /auth/token and the CLI (HTTP Basic) work with it.
    • d. Extra §7.6 check: adm_dohertj2 is refused at /auth/login, at /auth/token and on the management/CLI path (#145).
  7. OtOpcUa AdminUI, §7.5. §6 as Security__Ldap__* with FallbackServers__0 (the #913 build honours it), the §4.3 DEV GroupToRole, then Security__Auth__DisableLogin=false. Restart the single node. §7.6 plus acceptance A2 (nam1 blocked → login through nam2, with the library's sticky preference moved line). adm_dohertj2 is refused at /auth/login and /auth/token.
  8. Record the run as docs/deploy/2026-11-<dd>-vd03-nam-ldap-cutover.md in scadaproj, plus a line in each app's deployment notes. The mxgw notes must say "delete GwAdmin/GwReader from the base file on every redeploy". Tick the DEV items on scadaproj#6.

Stop rule (§7.6 rule 6) for this run: the account is Ignition's, so stop at the first service-bind warning, not the second. Ask IT for the account's lockout state before any retry.

Soak: a few working days of real logins on DEV.


0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19; EXECUTED 2026-11-19)

Executed 2026-11-19 15:29–16:10 EST: all four apps on both boxes on the DEV builds and NAM AD; record docs/deploy/2026-11-19-qa-nam-ldap/. Optional step 8 (oswaltk) not run.

Owner decisions (2026-11-19):

  1. QA binds as svc_igntadmin as well (interim). This supersedes A6 and the §7 note "QA never runs on svc_igntadmin". Both tiers move to the scadaproj#58 account together when it arrives: re-seed nam-ldap-svc-password in every store, then a rolling restart per app.
  2. QA gets exactly the builds DEV runs.
  3. One window: each app's code deploy and its LDAP change share one restart.

Gate: the owner lifts the QA hold and names the window. Tell the Ignition team before the window: QA adds 8 more stores that reference their admin account.

0Q.1 What QA holds today

Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP).

DEV-proven groups QA members (2026-11-19)
UG_WW_APP_ADMIN_QA (6) chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas
UG_WW_APP_IDE_QA (6) the same 6
UG_WW_APP_IDE_RO_QA (1) oswaltk (Viewer only, so a live Viewer test is possible with them)
UG_WW_HIST_ADMIN_QA (6) the same 6 as APP_ADMIN_QA, so the §0.2 Deployer interim carries over
UG_WW_BTMGR_RO_QA (0) empty; mapped per §4 anyway (grants nobody anything)

Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP. Anonymous GET / returns 302 → /login on :5220, :5130, :8085 and :9000. So QA is a directory swap, not a "turn login on". The GLAuth multi-role automation identity stops working at the cutover; the D3 interim is dohertj2, as on DEV.

App QA build now → target (= DEV) Where its LDAP config lives (QA) Notes
HistorianGateway (per box) 6365f34 f84af29 (#136) NSSM env Ldap__*, Dashboard__GroupToRole__SCADA-* [I] app\appsettings.json has Dashboard:RequireHttpsCookie=true, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below)
MxAccessGateway (per box) 76a71bc8 e74918cc app\appsettings.json: GLAuth localhost:3893, dc=zb,dc=local, ${secret:ldap/mxgateway/bind}, base GwAdmin/GwReader Phase 0 records the environment name and any overlay. Delete GwAdmin/GwReader from the base file (§4.2)
ScadaBridge (pair) 061432c8 2e0ce81e (#144/#145) app\appsettings.Central.json (GLAuth 127.0.0.1:3893, dc=scadabridge) + Central env password Secrets:Store:Provider=SqlServer: one shared store, so seed once. RequireHttpsCookie=false already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other
OtOpcUa (Warm pair) 76e73e7d cdf433a3 (#913; 83 commits, no migration) service env [I] Warm redundancy (Mode=Warm); Phase 0 records which node is Primary. Needs the cookie decision (finding 1)

Secrets master keys: the shipped files say Source=Environment (ZB_SECRETS_MASTER_KEY) for HG, mxgw and OtOpcUa. The service env may override that. The seed script reads each service's own env, so it never assumes.

0Q.2 Findings carried from DEV

  1. Plain-HTTP cookies. OtOpcUa's auth cookie is Secure by default, so its AdminUI login loops over HTTP. On DEV the fix was Security__Cookie__RequireHttpsCookie=false. QA OtOpcUa login works today over HTTP, so Phase 0 checks whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HG Dashboard:RequireHttpsCookie the same way.
  2. Secrets CLI. The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store schema or envelope change), copied to E:\ApiInstall\_tools\secret-0.6.6\ on each box.
  3. Do not add firewall rules or run DC-failover drills (owner, 2026-11-19).
  4. Logs: the apps' own files may be empty. Verification reads the box OTel collector export (E:\ApiInstall\OtelCollector\data\logs.jsonl) over SMB.

0Q.3 Mappings (QA suffix)

As §4, plus the Deployer interim:

Group OtOpcUa ScadaBridge MxAccessGateway HistorianGateway
UG_WW_APP_ADMIN_QA Administrator Administrator Administrator Administrator
UG_WW_APP_IDE_QA Designer Designer Viewer —
UG_WW_APP_IDE_RO_QA Viewer Viewer Viewer Viewer
UG_WW_HIST_ADMIN_QA Viewer Deployer (interim) — Administrator
UG_WW_BTMGR_RO_QA — Viewer — Viewer

Remove every GLAuth mapping: HG SCADA-* env entries and mxgw GwAdmin/GwReader. The ScadaBridge SCADA-* seed rows stay; they grant nothing in NAM.

0Q.4 The window

All remote steps are winrm deploy scope on vq03/vq04, one app at a time, in the canary order. Stop at the first service-bind warning (§0.4 stop rule). Each app's code deploy and LDAP change share one restart.

  1. Before the window (from the Mac, no remote exec):

    • stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB;
    • add the ScadaBridge role rows with the CLI while QA still logs in through GLAuth. Log in as multi-role, with --url pointing at the active central. This path is audited and invalidates the cache, unlike the SQL path DEV had to use. Add the five rows from §0Q.3; they grant nothing until the cutover.
  2. Phase 0 per box (preflight.ps1 from the DEV record, retargeted to the QA layout E:\ApiInstall\<App>\app and the service names):

    • DC reach on 636 and zimmer-rootca trust;
    • back up every service key and every appsettings*.json to E:\ApiInstall\_ldapcut-<ts>\;
    • record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa Secrets:Replication:Enabled, the OtOpcUa Primary/Standby and ServiceLevels, and the LdapGroupRoleMapping overlay rows;
    • read each app's secrets provider and master-key source from its service env.
  3. Seed nam-ldap-svc-password (seed-secret.ps1, generalised to read each service's own Secrets__* and master-key env):

    • HG and mxgw: per box;
    • OtOpcUa: per box, or once if replication is on;
    • ScadaBridge: once, into the shared SQL store, using the active Central's provider settings.

    Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted.

  4. HistorianGateway, vq03 then vq04 (independent single nodes; this is the canary):

    • deploy f84af29 with HG's own procedure (HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/ retargeted to QA);
    • before the start, rewrite the NSSM env: Ldap__* → §6, Dashboard__GroupToRole__* → §0Q.3, and the cookie per finding 1.

    Verify on each box: bogus refused, dohertj2 → Administrator, no system-side lines. Then the next box.

  5. MxAccessGateway, vq03 then vq04:

    • deploy e74918cc with mxgw's procedure. It will replace app\appsettings.json, so apply the edit to the new file before the start: §6 block, QA GroupToRole, delete GwAdmin/GwReader, no GLAuth pin;
    • mind the nssm START_PENDING abort in the repo upgrade script (memory note: use the try/catch fix as on vd03).

    Verify per box as in step 3.

  6. ScadaBridge pair: deploy 2e0ce81e with the QA pair procedure (deploy-prep/2026-11-15-143/qa.tmpl.ps1 actions minus migrate):

    • stop both Sites, then both Centrals, so the pair goes down together;
    • swap the binaries and edit each Central's appsettings.Central.json: §6 block, drop any Auth block;
    • remove ScadaBridge__Security__Ldap__ServiceAccountPassword from both Central envs;
    • start the Centrals (vq03 first), then the Sites.

    Verify:

    • dohertj2 gets Administrator+Designer+Deployer at /auth/login, at /auth/token and on the CLI against the active central;
    • bogus refused;
    • LocalDb sync connected; Z28062–Z28069 at their Good baseline; WarmCanary connected; one active per pair.
  7. OtOpcUa Warm pair: deploy cdf433a3 with OtOpcUa's QA procedure (the 2026-11-13 76e73e7d pack: stop-both / start-both for a binary upgrade, worker isolation -Reapply/-Verify).

    • Before the start, add to both nodes' env: the 18 Security__Ldap__* entries from the DEV record (QA suffix in GroupToRole), Security__Auth__DisableLogin left false, and the cookie per finding 1.
    • Verify:
      • ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads BadOutOfService;
      • LocalDb sync connected; the script-runner canary Good;
      • dohertj2 → Viewer+Designer+Administrator via cookie and /auth/token; bogus refused;
      • the ScadaBridge WarmCanary reconnects.
  8. Re-point QA automation from multi-role to dohertj2: the qa pack scripts, drills, apply.sh.

  9. Optional: oswaltk's Viewer-only login (the only Viewer-only member).

  10. Record in docs/deploy/2026-11-<dd>-qa-nam-ldap/, plus a deploy record in each repo.

Outages: HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is also a binary upgrade, and the pairs can't roll those.

Rollback: per app, restore the _ldapcut-<ts> keys and files. Binaries roll back with each repo's own procedure (no app here has a schema change). Keep the ScadaBridge UG_WW_* rows. The box-local GLAuth stays running until the soak ends.

Goal: move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair.

  • The service-account password lives as a ${secret:} reference in each app's encrypted Secrets store, never in config.
  • Every app maps AD groups to its roles from one canonical matrix, ldap_mapping.md, under two family rules (§3):
    • one AD group grants exactly one role per app;
    • a user whose groups map to no role cannot sign in (cookie or token).

In scope: HistorianGateway (dashboard), MxAccessGateway (dashboard), ScadaBridge (Central UI + CLI), OtOpcUa (AdminUI).

Out of scope:

  • Overview: no login by design.
  • The machine-to-machine surfaces stay API-key based: OtOpcUa's OPC UA data plane, the mxgw and HG gRPC APIs, and ScadaBridge's inbound API. They can still be disrupted by the restarts here (§7.0).
  • ignitionoee: on hold (ldap_mapping.md §6).

Evidence key: [V] = verified in source or a file, with the citation; [I] = inferred, to be confirmed in Phase 0. Source paths are relative to each repo.


1. Current state (2026-11-15)

1.1 Builds and the Auth library

The 2026-08-13 plan's "redeploy to Auth 0.2.1" phase is obsolete:

App Deployed on DEV and QA Auth pin [V]
OtOpcUa 76e73e7d 0.4.1 (Directory.Packages.props:238-240)
ScadaBridge 061432c8 0.5.0 (ScadaBridge#46)
MxAccessGateway 76a71bc8 0.4.1
HistorianGateway 6365f34 0.4.1

Domain-root SearchBase (≥ 0.2.1), FallbackServers (≥ 0.2.0 in the library, but dropped by OtOpcUa, §2) and BindMode with the empty-password guard (≥ 0.4.1) are all available.

Before the cutover:

  • OtOpcUa (§2) and ScadaBridge (§2A) need code changes and a deploy.
  • HistorianGateway (§2B) gets an optional hardening change.

1.2 Login state per app and tier

App DEV vd03 QA vq03/vq04
HistorianGateway Login OFF: Dashboard:DisableLogin=true in appsettings.Production.json. LDAP points at the shared GLAuth 10.100.0.35:3893, plain [V, file]. Login ON against the box-local GLAuth. Config is in the NSSM env (Ldap__*, GroupToRole SCADA-Admins/SCADA-Designers) [I]. KEK at C:\ProgramData\ZB\kek\historiangw.kek [V, 11-13 evidence].
MxAccessGateway Login OFF: MxGateway:Dashboard:DisableLogin=true [V, file]. LDAP 127.0.0.1:3893, dc=scadalink,dc=local. The deployed build refuses DisableLogin=true under Production (GatewayOptionsValidator.cs:484-488), so vd03 mxgw is not running as Production [V by inference from it booting]; Phase 0 records the environment name. Login ON: localhost:3893, dc=zb,dc=local, ${secret:ldap/mxgateway/bind} [V, file]. ⚠ deployment.md says the box-local GLAuth is dc=scadabridge,dc=local [I].
ScadaBridge Login OFF: ScadaBridge:Security:Auth:DisableLogin=true + AllowOutsideDevelopment=true [V, file]. With login off, the management/CLI path authenticates every caller as the all-roles dev user (ManagementService/ManagementAuthenticator.cs:63-71: "multi-role", Roles.All) [V]. LDAP localhost:3893. Login ON, box-local GLAuth 127.0.0.1:3893 [V]. Every CLI, deploy and drill script logs in as GLAuth multi-role.
OtOpcUa (AdminUI) Code defaults (Enabled=true, Server=localhost, Port=3893, SearchBase=dc=zb,dc=local, ServiceAccountDn=""), plus files holding Transport=Ldaps and DevStubMode=false; no Security__Ldap__* in the env [V, 11-13 record]. Security__Auth__DisableLogin is in the env, value unknown. Unknown; Phase 0 reads it.

1.3 Directory facts

Still valid, from ldap_setup.md:

  • DCs: uswap-vdc-nam1.nam.zimmer.com (10.201.216.2) and uswap-vdc-nam2.nam.zimmer.com (10.201.138.78).
  • Transports: LDAPS 636, StartTLS and plain 389 all work.
  • TLS: the certs are RSA-1024, chain leaf ← zimmer-issueca1 ← zimmer-rootca. The only chain error is UntrustedRoot, so installing the root is sufficient. Windows accepts the 1024-bit key.
  • Directory shape: SearchBase DC=nam,DC=zimmer,DC=com; login attribute sAMAccountName; memberOf collapses to bare CNs.
  • Reachability: from the Mac, nam1/nam2 on 389 and 636 are open (2026-11-15). Not proven from the boxes.

Login names. In BindMode=ServiceAccount (this plan), AcceptDomainQualifiedNames has no effect: it applies only in User mode (ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs:156-161) [V]. So NAM\jdoe and jdoe@nam.zimmer.com return user not found. Users must type their bare sAMAccountName. Put that on the login-page notice.

Lockout (AD policy: 5 failed binds per 15 min, per account) [V, LdapAuthService.cs]:

Failure Bad binds Effect
Wrong user password 1, on one DC (:164-168, :428-436) Final (BadCredentials); no walk to the next DC. Locks only that user, after 5 tries.
Wrong service-account password 2 per login attempt: system-side, so it walks to the next DC (:173-174, :408-415) Locks the shared account.

The shared account is one account for all four apps and both tiers (D1). A mistyped secret in any of the ~7 stores can therefore lock out login everywhere within a few attempts.

ScadaBridge's background group refresh also binds as this account. Its throttle is 5 per 5 min (ScadaBridge Security/SecurityOptions.cs:132-145) [V], looser than AD's 5 per 15 min. Mitigations:

  • D1 asks IT for a no-lockout (or very high threshold) policy on the account.
  • §7.1 verifies every store's value before that app restarts.
  • §7 does one canary app first.
  • §7.6 has a stop rule.

2. Phase A — OtOpcUa code fix (prerequisite for the OtOpcUa cutover)

One OtOpcUa change, one issue, one deploy. It fixes five things.

2.1 Defects [V]

  1. Options dropped. LdapOptions.ToLibraryOptions() (src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs:111) copies 12 fields into the library's LdapOptions (ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs:21).
    • Dropped: FallbackServers (0.2.0), BindMode, UserBindTemplate and AcceptDomainQualifiedNames (0.4.0). The config keys are ignored without error, so the AdminUI has no DC failover.
    • Not config-bound, by design: ServerCertificateValidationCallback, which is code-only.
  2. Zero-role sign-in (review B1). Security/Endpoints/AuthEndpoints.cs:124-158 signs in with an empty role set when the mapper returns no roles, and when the mapper throws: it catches the exception, logs a warning and continues with the baseline roles, which are empty on the real path.
    • The fallback policy and AdminUiPolicies.AuthenticatedRead (Security/ServiceCollectionExtensions.cs:192,217) only require an authenticated user, and 18 AdminUI pages use AuthenticatedRead.
    • No policy requires Viewer; it is only a default in RoleGrants.razor:126.
    • With a domain-root SearchBase, any NAM user with any group would get AdminUI read access, and /auth/token would mint them a token.
  3. No library logger (review M1). OtOpcUaLdapAuthService.cs:48 builds LibLdapAuthService with no logger, and the library logs only via _logger?.. So the failover log line (sticky preference moved) and the stage diagnostics (failed system-side at …) never appear in OtOpcUa, and §7.6's checks can't be done.
  4. No validation of the new options (review B2). OtOpcUa never runs the library's LdapOptionsValidator: it constructs the library service directly (OtOpcUaLdapAuthService.cs:48). That answers the old open question.
    • Calling the library validator would stop OtOpcUa booting on DEV and QA. It requires ServiceAccountDn in ServiceAccount mode (ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs:81-87), DEV runs on code defaults (Enabled=true, ServiceAccountDn=""), and OtOpcUa validates LDAP on every node (Host/Program.cs:190-195).
  5. Stale docs. The OtOpcUa validator comment (Host/Configuration/LdapOptionsValidator.cs:12-14) describes a "direct-bind path" that doesn't exist. The LdapOptions doc says the default server is 10.100.0.35; the code says localhost.

2.2 The fix

  1. Add the four properties with the library's defaults and pass them through ToLibraryOptions():

    • List<string> FallbackServers = [] → IReadOnlyList<string>
    • LdapBindMode BindMode = ServiceAccount
    • string UserBindTemplate = ""
    • bool AcceptDomainQualifiedNames = true
  2. Refuse a zero-role login (the family standard, §3.2). In /auth/login and /auth/token:

    • if the mapped role set is empty, refuse exactly like bad credentials (HTTP 401 / the login page's "invalid or not authorized"), with no cookie and no token;
    • if the mapper throws, also refuse (fail closed), log the exception, and do not sign in.
    • This reverses the current code comment's intent and the test Login_when_role_mapper_throws_signs_in_with_no_role_claims (tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/AuthEndpointsIntegrationTests.cs:402). Rewrite that test to assert refusal.
    • Add tests: empty mapping refused (cookie + token); mapper throws refused (cookie + token); one mapped role signs in.
    • DevStub is unaffected: it grants Administrator.
  3. Inject the logger: pass ILogger<ZB.MOM.WW.Auth.Ldap.LdapAuthService> from DI into the LibLdapAuthService constructor at OtOpcUaLdapAuthService.cs:48. Add a test that a failover writes the library's sticky preference moved line.

  4. Validation, OtOpcUa's own validator only. In Host/Configuration/LdapOptionsValidator.cs, port only:

    • the library's FallbackServers rule (each entry parses as host[:port]);
    • the BindMode=User → UserBindTemplate contains {0} exactly once rule.

    Do not port the ServiceAccountDn-required rule.

    Add a test that boots the validator with today's DEV-shaped config (Enabled=true, Server=localhost, ServiceAccountDn="", Transport=Ldaps) and asserts it passes.

  5. Completeness guard. Add a unit test that reflects over the library LdapOptions' public settable properties.

    • For each one, set a non-default value on OtOpcUa's options, call ToLibraryOptions(), and assert the value arrived. Compare IReadOnlyList values by content.
    • Check both directions: every library property is mapped or allow-listed, and no allow-list entry is stale.
    • Allow-list: ServerCertificateValidationCallback.
    • Mutation check: delete one assignment and the test must fail.
  6. Binding test: Security:Ldap:FallbackServers:0 reaches the library options, in JSON and as the env var Security__Ldap__FallbackServers__0.

  7. Docs:

    • docs/Configuration.md Security:Ldap: the new keys, a NAM two-DC example, the zero-role refusal, and "type the bare sAMAccountName".
    • Fix the two stale comments from §2.1 item 5.
  8. Acceptance A1 (offline): an AdminUI login against the redundant GLAuth pair (infra/glauth-redundant/, :3893 + FallbackServers ["…:3894"]).

    • Stop the primary: login still succeeds and the log shows sticky preference moved to …:3894.
    • A GLAuth user in no mapped group is refused.

2.3 Issue text for lmxopcua (not filed by this plan)

Title: AdminUI LDAP: drops FallbackServers/BindMode/…, signs in users with no mapped role, and the library logs nothing

Labels: type:bug, priority:p1, area:auth, cross-repo

Body:

  1. LdapOptions.ToLibraryOptions() (Security/Ldap/LdapOptions.cs:111) drops FallbackServers, BindMode, UserBindTemplate and AcceptDomainQualifiedNames, so there is no AD DC failover.
  2. AuthEndpoints.cs:124-158 signs in with an empty role set, including when the mapper throws, and AuthenticatedRead admits any authenticated user. Once the search base is a whole AD domain, any directory user gets AdminUI read access and a token. The fix refuses empty roles at /auth/login and /auth/token, and refuses when the mapper throws. This reverses Login_when_role_mapper_throws_signs_in_with_no_role_claims.
  3. OtOpcUaLdapAuthService.cs:48 passes no logger to the library, so failover and stage logs are missing. Inject ILogger<LdapAuthService>.
  4. Validate the new options in OtOpcUa's own validator: the fallback parse rule and the BindMode/UserBindTemplate rules only. Not ServiceAccountDn, because DEV boots on code defaults with an empty DN.
  5. A two-way reflection completeness test, a binding test, docs, and fixes to the stale comments.

Acceptance: failover on the redundant GLAuth pair, a zero-role user refused (cookie + token), and a DEV-shaped config boots. Plan: scadaproj/docs/plans/2026-11-15-nam-ldap-cutover.md §2.

2.4 Ship it

Merge to master and deploy with OtOpcUa's own procedure (docs/deploy/, most recently 2026-11-13-vd03-qa-to-76e73e7d.md): DEV, then QA.

  • No LDAP config change goes out with this deploy.
  • The zero-role refusal does change behaviour on QA: it is live against the box-local GLAuth today, so any GLAuth user without a mapped group loses AdminUI read access. Phase 0 checks that the multi-role user maps.
  • Gate: §7.5 (the OtOpcUa cutover) does not start until the deployed build contains this fix.

2A. Phase A2 — ScadaBridge zero-role refusal (prerequisite for the ScadaBridge cutover)

Defect [V, review B1]:

  • CentralUI/Auth/AuthEndpoints.cs:59-115 signs in (cookie) and issues a bearer at /auth/token (:125) even when RoleMapper returns no roles.
  • Dashboard.razor:2 and Monitoring/Health.razor:2 are bare [Authorize], so any directory user could read them.

Fix:

  1. In /auth/login and /auth/token, an empty mapped role set returns the bad-credentials response, with no cookie and no token.
  2. Apply the same rule to the mid-session group refresh (DirectoryGroupRefresher). A session whose roles drop to zero is already ended by that refresh (:141-142); keep that behaviour and test it.
  3. Tests: zero roles refused (login + token); one role signs in; a refresh to zero ends the session.
  4. Docs: the Component-Security / CentralUI auth section.
  5. Leave the bare [Authorize] pages as they are; they are safe once a login requires a role. Add a static test that every page is either policy-gated or relies on that rule.

Issue text for ScadaBridge (not filed):

Title: Central UI / /auth/token sign in users whose AD groups map to no role

Labels: type:bug, priority:p1, area:auth, cross-repo

Body: CentralUI/Auth/AuthEndpoints.cs:59-115 signs in and mints a bearer with an empty role set. With a domain-wide NAM search base, any AD user reaches Dashboard and Monitoring/Health (bare [Authorize]). The family standard (scadaproj NAM cutover plan §3.2) refuses login when no role is mapped. Fix login and token, keep the refresh's end-on-zero, and add tests.

Acceptance: a zero-role user is refused at both endpoints. Plan: docs/plans/2026-11-15-nam-ldap-cutover.md §2A.

Ship: a ScadaBridge build containing the fix, deployed DEV then QA before §7.4. This is a binary deploy, so it follows the QA pair rules: stop both centrals and start them together.


2B. Phase A3 — HistorianGateway case-insensitive group match (optional hardening)

Defect [review M5]: Dashboard/DashboardOptions.cs:44-45 declares IReadOnlyDictionary<string,string> with an OrdinalIgnoreCase default. The configuration binder replaces it with a new case-sensitive Dictionary. The reviewer proved this with an offline binder test; the XML doc says "compared case-insensitively", which is wrong once it's bound. AD group names are mixed case, so a key whose case differs from the memberOf CN never matches.

Fix: declare Dictionary<string,string> with StringComparer.OrdinalIgnoreCase, so the binder fills the existing instance, as mxgw and OtOpcUa do. Add a binder test with a mixed-case key.

Until it ships (and in any case), §4 copies each group name exactly as LdapProbe prints it.

Issue text for historiangw (not filed):

Title: Dashboard:GroupToRole is case-sensitive after binding (docs say case-insensitive)

Labels: type:bug, priority:p3, area:auth

Body: the IReadOnlyDictionary default with OrdinalIgnoreCase is replaced by the binder's default Dictionary. Make the property a Dictionary with OrdinalIgnoreCase and add a binder test with a mixed-case key. Found by the NAM cutover review (scadaproj plan §2B).


3. The family standards (DECIDED 2026-11-15)

3.1 One AD group grants exactly one role per app

A user's roles in an app are the union of the roles of every mapped group they belong to. No app maps one group to two roles. Each deployment maps only its own _DEV/_QA/_PROD suffix.

Why:

  1. Every schema already enforces or implies it, so no code change is needed:

    App Mapping shape [V]
    OtOpcUa Dictionary<string,string> GroupToRole (Security/Ldap/LdapOptions.cs:84), unioned with the DB overlay LdapGroupRoleMapping (OtOpcUaGroupRoleMapper.cs). The overlay's only unique index is (LdapGroup, ClusterId), filtered, so two system-wide rows per group are possible; the rule forbids them and Phase 0 checks.
    ScadaBridge table LdapGroupMappings, with HasIndex(LdapGroupName).IsUnique() (ConfigurationDatabase/Configurations/SecurityConfiguration.cs:26). RoleMapper unions roles across different groups (Security/RoleMapper.cs:48-60).
    MxAccessGateway Dictionary<string,string> (Configuration/DashboardOptions.cs:69). The validator allows only Administrator/Viewer (GatewayOptionsValidator.cs:506-517).
    HistorianGateway IReadOnlyDictionary<string,string> (Dashboard/DashboardOptions.cs:44). Case-sensitive in practice (§2B).
  2. Grants stay readable in AD, where IT owns membership.

  3. It removes a silent failure: the old two-row ScadaBridge insert hits the unique index.

Rejected: several roles per group. It would need a ScadaBridge migration and matching changes in the other apps. It stays a future option.

3.2 No role, no login (added by review B1)

Every app refuses sign-in, both cookie and token, when the user's groups map to no role. That includes the case where the role lookup fails.

App Before the fixes [V] Change (status 2026-11-17: merged, not deployed)
MxAccessGateway refuses (Dashboard/DashboardAuthenticator.cs:66-70) none
HistorianGateway refused zero roles; a throwing mapper escaped as a 500 historiangw#136, f84af29: case-insensitive GroupToRole, a mapper fault is refused like a bad password, both logged at Warning (Dashboard/DashboardAuthenticator.cs, the MapAsync catch and the roles.Count == 0 branch)
OtOpcUa signed in with no roles, even when the mapper threw §2 (Phase A): lmxopcua#913, cdf433a35
ScadaBridge signed in with no roles: cookie, /auth/token, management HTTP Basic/Bearer §2A (Phase A2): #144 d48e1667 (cookie, token, session refresh) + #145 2e0ce81e (management/CLI, audit, debug stream)

Consequence for the matrix: read access now requires a mapped group. In OtOpcUa, Viewer is not checked by any policy (pages use AuthenticatedRead), so a Viewer cell means "holds a role, so may sign in and read". That is why APP_IDE_RO_<T> → Viewer stays, and why HIST_ADMIN_<T> → Viewer in OtOpcUa is meaningful.

3.3 Consequence for ScadaBridge Deployer

APP_IDE → Designer only. In ScadaBridge, Administrator does not imply Deployer: each policy requires its own role (Security/AuthorizationPolicies.cs:190-202). No existing group means "deploy", so:

  • a new AD group UG_WW_APP_DEPLOY_{DEV,QA} (D6) maps to Deployer;
  • until it exists on a tier, the ScadaBridge cutover does not run there, because nobody could deploy.

The other apps have no Deployer role; OtOpcUa deploy is ConfigEditor = Administrator/Designer, AdminUiPolicies.cs:18.


4. The tier mappings (generated from ldap_mapping.md §4.1)

These are platform groups only. Plant-floor, site and batch groups are not in this cutover (D5), and CITRIX_* is never mapped. Copy each group name exactly as LdapProbe prints it (§2B; D7). The names below are as observed on 2026-08-12.

Group (<T> = DEV or QA) OtOpcUa ScadaBridge MxAccessGateway HistorianGateway
UG_WW_APP_ADMIN_<T> Administrator Administrator Administrator Administrator
UG_WW_APP_IDE_<T> Designer Designer Viewer —
UG_WW_APP_DEPLOY_<T> (new, D6; only once it exists) — Deployer — —
UG_WW_APP_IDE_RO_<T> Viewer Viewer Viewer Viewer
UG_WW_HIST_ADMIN_<T> Viewer — — Administrator
UG_WW_BTMGR_RO_QA (QA only; empty as of 08-12) — Viewer — Viewer

4.1 HistorianGateway (DEV: appsettings.Production.json; QA: NSSM env)

"Ldap": { /* §6 */ },
"Dashboard": {
  "DisableLogin": false,
  "GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_HIST_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
}

On QA, remove every Dashboard__GroupToRole__SCADA-* entry first, then set:

Dashboard__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_HIST_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer
Dashboard__GroupToRole__UG_WW_BTMGR_RO_QA=Viewer

4.2 MxAccessGateway (the overlay file for the box's environment name; Phase 0 records it)

The base appsettings.json ships GwAdmin/GwReader mappings (:86-89) [V], and dictionaries merge across overlays. So the overlay alone can't remove them, and they can't be neutralised either, because the validator accepts only Administrator/Viewer as a value. Delete the two entries from the box's base appsettings.json (backed up). A redeploy restores the base file, so the mxgw deploy notes must carry the deletion forward.

Then set:

"MxGateway": {
  "Ldap": { /* §6 */ },
  "Dashboard": {
    "DisableLogin": false,
    "GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_DEV": "Viewer", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
  }
}

On QA, use the _QA suffix and delete GwAdmin/GwReader the same way. Acceptance A4 reads the effective config (§8), not just the edited file.

4.3 OtOpcUa (service env, both tiers)

Security__Ldap__GroupToRole__UG_WW_APP_ADMIN_<T>=Administrator
Security__Ldap__GroupToRole__UG_WW_APP_IDE_<T>=Designer
Security__Ldap__GroupToRole__UG_WW_APP_IDE_RO_<T>=Viewer
Security__Ldap__GroupToRole__UG_WW_HIST_ADMIN_<T>=Viewer
  • DB overlay: the RoleGrants page / LdapGroupRoleMapping must hold no rows for these groups, or only the same role.
  • DriverOperator also honours the role Operator (ServiceCollectionExtensions.cs:201-202); nothing maps to it here.

4.4 ScadaBridge (central database)

Preferred: the management surface, so the change is audited by the ManagementActor and the 30 s cross-central role-mapping cache is invalidated.

  • Use the CLI security role-mapping commands: scadabridge security --help lists them. Run them on the target tier with an explicit --url, while DEV login is still off (the dev user holds every role).
  • On QA, run them as the current GLAuth admin before the cutover.
  • The Central UI's role-mapping page is equivalent.

Add one mapping per group:

  • UG_WW_APP_ADMIN_<T> → Administrator
  • UG_WW_APP_IDE_<T> → Designer
  • UG_WW_APP_IDE_RO_<T> → Viewer
  • UG_WW_APP_DEPLOY_<T> → Deployer, only if that group exists (D6)
  • QA only: UG_WW_BTMGR_RO_QA → Viewer

Fallback: direct SQL (idempotent; refuses to change a conflicting row). It bypasses the audit and the cache invalidation, so record it in the deployment record and wait 30 s (or restart Central) before testing.

SET XACT_ABORT ON;
BEGIN TRAN;
DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL);
INSERT @t VALUES (N'UG_WW_APP_ADMIN_DEV', N'Administrator'), (N'UG_WW_APP_IDE_DEV', N'Designer'), (N'UG_WW_APP_IDE_RO_DEV', N'Viewer');
-- Only if D6's group exists:  INSERT @t VALUES (N'UG_WW_APP_DEPLOY_DEV', N'Deployer');
-- QA: _QA suffix, plus (N'UG_WW_BTMGR_RO_QA', N'Viewer').
IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role)
    THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1;
INSERT dbo.LdapGroupMappings (LdapGroupName, Role)
SELECT t.LdapGroupName, t.Role FROM @t t
WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName);
COMMIT;
SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id;

Names [V]:

  • Table LdapGroupMappings.
  • Columns LdapGroupName nvarchar(500) and Role nvarchar(100) (SecurityConfiguration.cs:16-24).
  • Role values are the Security/Roles.cs:45-56 constants.

The SCADA-* seed rows (Ids 1–5) name GLAuth groups that don't exist in NAM, so they grant nothing; leave them.


5. Decisions (owner or IT): settle before the cutover (§7)

# Decision Options Recommendation
D1 Service account (owner request: one dedicated LDAP-lookup account shared by the four services, scadaproj#58) Target: the dedicated account from scadaproj#58, shared across all four apps; one account for DEV and QA unless IT prefers one per tier. Interim (DEV only): svc_igntadmin@nam.zimmer.com, only with the Ignition team's agreement. It is their admin account, and their rotations or lockouts would hit us. Target from scadaproj#58, a QA prerequisite. Add to the scadaproj#58 ask: a no-lockout policy (or a very high threshold) on the account, because it is shared by four apps and two tiers and a wrong secret costs 2 bad binds per login (§1.3). BindMode=ServiceAccount everywhere. Rotation = re-seed nam-ldap-svc-password in every store and restart each app, since values resolve only at startup (§7.1).
D2 Turn login on where DEV is deliberately open (HG; mxgw = mxaccessgw#231 item 3; ScadaBridge) Turn it on / leave DEV open Turn it on. DEV is where problems should surface.
D3 Automation identity (CLI, deploy and drill scripts). QA: today GLAuth multi-role. DEV: today the login-off dev user (ManagementAuthenticator.cs:63-71), which stops working the moment DEV ScadaBridge login turns on. The same applies to OtOpcUa scripts if DEV Security__Auth__DisableLogin=true (Phase 0). (a) A NAM user-type identity per tier in APP_ADMIN_<T> + APP_IDE_<T> + APP_DEPLOY_<T>, password in the Mac Keychain. The scadaproj#58 account is lookup-only and should not double as a login unless IT agrees. (b) An interim DEV identity (a named admin's account) for DEV only. (a). A precondition of the DEV ScadaBridge step (or (b) as a DEV interim) and of the QA ScadaBridge step. Also of the OtOpcUa step if Phase 0 finds DEV OtOpcUa login off.
D4 TLS trust (a) Install zimmer-rootca in LocalMachine\Root, AllowInsecure=false. (b) AllowInsecure=true: accepts any certificate (ZB.MOM.WW.Auth.Ldap/Internal/NovellLdapConnection.cs:41-46), so an active man-in-the-middle can read the service-account and every user's password. (c) Pin: in OtOpcUa, HG and ScadaBridge this is code-only (ServerCertificateValidationCallback), but mxgw has a config-bound strict pin, MxGateway:Ldap:ServerCertificateThumbprintSha256 (LdapServerCertificatePin.cs:9-21). (a). (b) is DEV only and short-lived, if at all. For mxgw: remove any GLAuth pin left in config (§7.3), and optionally pin the NAM DC leaf thumbprints. The certs expire 2027-03, so a pin must be renewed.
D5 Plant-floor and site groups now / later Later (ldap_mapping.md §6).
D6 New group UG_WW_APP_DEPLOY_{DEV,QA} (ScadaBridge Deployer) ticket it / the rejected multi-role code (§3.1) Ticket it. Blocks the ScadaBridge step on each tier.
D7 Group membership and exact names — Per person: dotnet run --project tools/LdapProbe -- --insecure --svc '<svc account UPN>' -u <sAMAccountName> (service-account path; prompts for the service password). Or, as the person, --insecure --self-bind -u <sAMAccountName> (binds as them; --svc is ignored). Families: -g 'UG_WW_*'. Missing people get a ServiceNow membership request, not a wider mapping. Copy group names exactly (§2B).

6. The LDAP block (each app's section)

The sections are: HG Ldap, mxgw MxGateway:Ldap, ScadaBridge ScadaBridge:Security:Ldap (Security/ServiceCollectionExtensions.cs:24), OtOpcUa Security:Ldap.

{
  "Enabled": true,
  "Server": "uswap-vdc-nam1.nam.zimmer.com",
  "Port": 636,
  "Transport": "Ldaps",
  "AllowInsecure": false,                                       // D4
  "FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ],   // OtOpcUa: only with the §2 build
  "SearchBase": "DC=nam,DC=zimmer,DC=com",
  "ServiceAccountDn": "<svc account UPN from scadaproj#58>",     // D1; DEV interim: svc_igntadmin@nam.zimmer.com (Ignition agreement)
  "ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
  "UserNameAttribute": "sAMAccountName",
  "DisplayNameAttribute": "name",
  "GroupAttribute": "memberOf",
  "BindMode": "ServiceAccount"
}

mxgw only: no ServerCertificateThumbprintSha256 unless D4 chooses to pin the NAM DCs.

The env form follows the usual pattern, e.g. Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636.


7. Runbook

Order:

  1. DEV: HistorianGateway (the canary, §7.6 rule 6) → MxAccessGateway → ScadaBridge (after §2A, D3, D6) → OtOpcUa (after §2).
  2. A soak of a few working days with real logins.
  3. QA in the same order. QA also needs scadaproj#58 (QA never runs on svc_igntadmin). If DEV used the interim account, switch DEV to the scadaproj#58 account during the soak, so that QA repeats a configuration already proven on DEV.

One app at a time: edit, restart, verify, then the next.

Remote execution only within the winrm deploy scope: preflight, backup, edit, restart, verify, rollback. Files go over SMB and SQL runs from the Mac. servecli is retired.

Restart rules (review M4). This is a config-only change: the same binary, no migration, no wire change. So restarts are rolling, standby first:

Pair How Why
OtOpcUa QA pair Restart the standby (ServiceLevel 240/90), confirm it is healthy and back at its level, then the primary. The primary's restart is a failover; ScadaBridge's OPC UA client follows ServiceLevel, as WarmCanary showed. The stop-both/start-both rule in OtOpcUa's records is for binary/schema upgrades. Config-only allows rolling [I: confirm against OtOpcUa's deploy notes in Phase 0]. A bad LDAP key stops a node booting, which is why the standby goes first.
ScadaBridge QA centrals Standby central first, confirm it rejoins (one active, /health/active), then the active one, which gives a takeover of about 8–11 s. Sites are not restarted. Central LDAP config only. The LocalDb stop-together rule applies to the site pairs' sync, which this change doesn't touch. The stop-together rule for binaries (§2A) does not apply.
Single nodes (DEV; HG/mxgw on each QA box) One restart each. —

A pair restarted together is an outage of its data plane (OPC UA or the site data path). Don't do it for this change.

7.0 Phase 0 — preflight (read-only; deploy scope; per box vd03, vq03, vq04)

  1. DC reachability: Test-NetConnection to nam1 and nam2 on 636 and 389.
  2. TLS: check the chain to nam1:636 (SslStream, or certutil -verify) and whether zimmer-rootca is in LocalMachine\Root. This settles D4.
  3. Service-account bind, once from the Mac: LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D '<svc account UPN>' -W -b "DC=nam,DC=zimmer,DC=com" -s base dn
  4. Current config per app, kept for rollback:
    • hash the appsettings;
    • export the NSSM/service env;
    • list every Ldap__*, *__Ldap__*, *DisableLogin*, *__GroupToRole__* entry;
    • record OtOpcUa's Security__Auth__DisableLogin on all three boxes;
    • record mxgw's environment name (it decides the overlay file; vd03 can't be Production while DisableLogin=true);
    • record mxgw's ServerCertificateThumbprintSha256 (a leftover GLAuth pin blocks NAM);
    • record mxgw's base GwAdmin/GwReader entries.
  5. QA mxgw dc=zb vs dc=scadabridge: does a test login work today? This decides the rollback target.
  6. Secrets per app:
    • the store path, master-key source and KEK location for the §7.1 fallback. The family default is MasterKey:Source=Environment / ZB_SECRETS_MASTER_KEY; file-based KEKs exist on some boxes. Record which each app uses; don't assume.
    • ScadaBridge Central Secrets:Store:Provider. QA is the shared SQL store (2026-11-13 QA pack §1); DEV is [I] Sqlite.
    • OtOpcUa Secrets:Replication:Enabled (Host/Configuration/SecretsRegistration.cs:21). If it's on, the QA pair shares the secret through Akka replication, and "one store per box" becomes "seed once".
    • HG KEK health on vd03 (historiangw#134).
  7. OtOpcUa DB overlay: the LdapGroupRoleMapping rows (§4.3).
  8. Builds:
    • the deployed OtOpcUa build contains §2 (required before §7.5);
    • the deployed ScadaBridge build contains §2A (before §7.4);
    • on QA, the GLAuth multi-role user still maps to a role after §2/§2A (§2.4).

7.1 Seed and verify the secret (before any config change)

  • Name: nam-ldap-svc-password, the same in every store.
    • Values resolve only at startup, so a rotation is re-seed every store plus restart every app (§7 restart rules).
    • An unresolved ${secret:} throws SecretNotFoundException at boot (SecretReferenceExpander.cs:74-77), so the app won't start.
  • Preferred: each app's /admin/secrets page. It writes through the app's own provider, which is correct for ScadaBridge's shared SQL store and for a replicated OtOpcUa store.
    • DEV HG, mxgw and ScadaBridge are still login-off at this point. On QA, use the current GLAuth admin.
  • Fallback: the secret CLI, with the store and master-key settings from Phase 0 step 6, per app:
    $env:Secrets__SqlitePath = "<that app's store, Phase 0>"
    $env:Secrets__MasterKey__Source = "<Environment|File, Phase 0>"   # family default: Environment (ZB_SECRETS_MASTER_KEY)
    $env:Secrets__MasterKey__FilePath = "<kek, only if File>"
    secret.exe set nam-ldap-svc-password   # prompts; never pass the value as an argument
    
  • Stores:
    • HG and mxgw: per box.
    • OtOpcUa: per box, or once if replication is on.
    • ScadaBridge: once per tier through Central. The site does no LDAP.
  • Verify the VALUE before the app's restart (review M2): read it back (get) and bind with it once using LdapProbe (--svc '<svc account UPN>', pasting the read-back value). One good bind proves the stored secret. A wrong secret found here costs 1–2 bad binds, instead of 2 per user login after the restart.

7.2 HistorianGateway (DEV: file; QA: NSSM env). The canary.

  1. Back up the file or env.
  2. Write §6 and §4.1. On DEV, set DisableLogin to false. On QA, replace the Ldap__* entries and remove the SCADA-* GroupToRole entries.
  3. Restart ZbHistorianGateway. Run §7.6 fully, including the failover check, before touching any other app.

7.3 MxAccessGateway (the file for its environment name)

  1. Back up the base and overlay files.
  2. Write MxGateway:Ldap (§6) and GroupToRole (§4.2), and delete the base GwAdmin/GwReader.
  3. Remove the ServerCertificateThumbprintSha256 pin, or replace it with the NAM DC leaf thumbprints (D4).
  4. On DEV, set DisableLogin to false (D2, mxaccessgw#231 item 3). If that makes the box eligible to run as Production, record the environment name. Don't change it in this step.
  5. Leave the API-key store under data\ and the pepper env alone.
  6. Restart, then run §7.6.

7.4 ScadaBridge (Central only)

Preconditions:

  • §2A deployed;
  • D6: UG_WW_APP_DEPLOY_<T> exists, has members, and is mapped;
  • D3: the automation identity works. On DEV it is needed before login turns on, because the dev-user path ends then.
  1. Back up appsettings.Central.json and the env.
  2. Add the role mappings (§4.4, CLI preferred).
  3. Write ScadaBridge:Security:Ldap (§6).
  4. On DEV, remove BOTH ScadaBridge:Security:Auth:DisableLogin and AllowOutsideDevelopment. Removing only the second while DisableLogin=true makes DisableLoginGuard crash-loop the service.
  5. On QA, replace the GLAuth values.
  6. Restart: DEV, the single central. QA, rolling, standby central first (§7 restart rules).
  7. Re-point every script that used multi-role (or relied on login-off) at the D3 identity: apply.sh, the drill and deploy packs. Confirm /auth/token works with it.

7.5 OtOpcUa AdminUI (service env; only after §2 is deployed)

  1. Back up the env.

  2. Add §6 as Security__Ldap__* and the §4.3 GroupToRole. Keep the files' Transport/DevStubMode.

  3. If Security__Auth__DisableLogin=true, set it to false, and if scripts relied on it, have D3 in place first.

  4. Restart: DEV, the single node. QA, rolling, standby first (§7 restart rules). Watch ServiceLevel and the ScadaBridge WarmCanary connection across the primary's restart.

  5. Existing sessions: AdminUI cookies and bearer tokens issued under GLAuth stay valid after the switch until they expire. Either:

    • invalidate them (restart both nodes, if the cookie and data-protection keys are per process [I], or rotate the signing or data-protection key); or
    • record their remaining lifetime and accept it.

    ScadaBridge needs no such step: its group refresh ends sessions whose roles change (DirectoryGroupRefresher.cs:141-142).

7.6 Verify (each app, right after its restart)

  1. Negative control first: a bogus user and password. Expect a clean "invalid / not authorized" and no failed system-side warning.
    • A service-bind-stage warning means the stored service secret is wrong. Stop: every further attempt costs the shared account 2 bad binds.
    • A connect-stage warning means network or TLS.
    • (An unresolved ${secret:} never gets this far; the app fails at boot with SecretNotFoundException.)
    • (OtOpcUa logs these lines only with the §2 logger fix.)
  2. One real login per mapped role: Administrator, Designer, Viewer, and Deployer for ScadaBridge. The UI shows exactly that role, and the user typed a bare sAMAccountName (§1.3).
  3. A user whose groups map to no role is refused, on the login page and at the token endpoint where one exists (OtOpcUa and ScadaBridge /auth/token). This is the §3.2 standard: in OtOpcUa and ScadaBridge it holds only with the §2/§2A builds; mxgw and HG already refuse.
  4. ${secret:} proof: no *ServiceAccountPassword sits in env, and the stored value was bind-verified (§7.1).
  5. Failover:
    • On HG (the canary), once per tier: block nam1:636 briefly. Login still works and the log shows sticky preference moved; then restore.
    • OtOpcUa gets its own nam1→nam2 check (acceptance A2).
  6. Stop rule (review M2): stop the cutover on 2 failed logins attributable to the service account, across all apps, within 15 minutes. Count service-bind warnings, not user typos. Diagnose before any retry; ask IT to check the account's lockout state.

7.7 Rollback (per app)

  • Restore the backed-up files or env and restart. On QA, use the same rolling order: standby first.
    • mxgw: also restore its base appsettings.json (the GwAdmin/GwReader entries) and any pin.
  • Point the scripts back at multi-role (QA), or back at login-off (DEV ScadaBridge).
  • If the shared account got locked, IT clears the lockout. Until then, every NAM-backed app is down for login; roll those apps back to GLAuth or login-off first.
  • The seeded secrets are inert once nothing references them. The ScadaBridge UG_WW_* mapping rows grant nothing while login is off or on GLAuth.
  • Keep the §2/§2A/§2B builds. The zero-role refusal is the family standard and stands on its own.

8. Acceptance

  • A1: the §2 offline tests pass:
    • failover on the redundant GLAuth pair with the library's log line;
    • zero-role refusal (cookie + token, including the mapper-throws case);
    • a DEV-shaped config boots;
    • the reflection guard fails under its mutation.
    • The §2A tests pass too.
  • A2: on each tier, the OtOpcUa AdminUI login survives nam1 being blocked, through nam2.
  • A3: every §7.6 check passes for each tier and app.
  • A4: no app maps any group to two roles, and no stale GLAuth mapping remains. Check the effective config: env plus all overlays plus base, including mxgw's GwAdmin/GwReader, and the ScadaBridge table.
  • A5: automation runs on the D3 identity. Nothing uses multi-role, or relies on login-off, on a cut-over app.
  • A6: QA (and DEV after the soak) binds as the scadaproj#58 account and never as svc_igntadmin.

9. Open questions

  1. Does OtOpcUa run the library's validator? Answered: no (§2.1 item 4). The validation is ported selectively (§2.2 item 4).
  2. Does QA mxgw login work today (dc=zb vs dc=scadabridge)? Phase 0 step 5.
  3. What is the HG KEK state on vd03? (historiangw#134)
  4. Is OtOpcUa's AdminUI cookie or data-protection key per process or shared? That decides the §7.5 step 5 method [I].
  5. Does OtOpcUa's deploy procedure allow a rolling restart for config-only changes? Confirm against its records (§7 restart rules).
  6. Do the UG_WW_* memberships and exact names still match 2026-08-12? D7 re-enumerates.
  7. ignitionoee: still on hold.

10. Corrections from the 2026-08-13 plan

08-13 plan said Fact (2026-11-15)
Phase 1: redeploy to Auth 0.2.1 Every build already runs 0.4.1 (ScadaBridge 0.5.0). Phase deleted.
Preflight over servecli :2222 servecli is retired; the winrm deploy scope plus SMB plus SQL replace it.
mxgw on vd03 has LDAP login on DisableLogin=true (and so it isn't running as Production).
One target block for all four apps OtOpcUa drops FallbackServers and three more options, and logs nothing. Fixed in §2.
A zero-group user is denied in every app Only mxgw and HG refuse. OtOpcUa and ScadaBridge sign in with no roles. Fixed in §2/§2A; the standard is §3.2.
ScadaBridge INSERT INTO LdapGroupMapping, two rows for APP_IDE The table is LdapGroupMappings, unique LdapGroupName. One-group-one-role plus the new D6 group; CLI preferred over SQL.
ldap_mapping.md: ScadaBridge can grant two roles per group It can't. Rewritten as the canonical matrix.
Seed the ScadaBridge central and site stores with the CLI Seed once through Central /admin/secrets; the site does no LDAP.
Stop after 2 failures; every failed login costs both DCs A wrong user password costs 1 bind. A wrong service secret costs 2 per login and locks the shared account. New stop rule, a no-lockout ask and value verification.
Service account svc_igntadmin now, dedicated "later" Target: scadaproj#58, a QA prerequisite; svc_igntadmin is a DEV interim only, with Ignition's agreement.
OtOpcUa before ScadaBridge OtOpcUa last, after its fix.
(silent) automation DEV and QA automation lose their identity on cutover. D3 is a precondition.
(silent) restarts Rolling, standby first, for config-only changes. Restarting a pair together is a data-plane outage.

11. Review 2026-11-15: findings and disposition

# Finding Disposition
B1 OtOpcUa and ScadaBridge sign in users with zero mapped roles (OtOpcUa even when the mapper throws); AuthenticatedRead/bare [Authorize] then grant read access to any AD user Applied. New standard §3.2; OtOpcUa fix §2.2 item 2 (refuses on throw too, reversing Login_when_role_mapper_throws_signs_in_with_no_role_claims); ScadaBridge Phase A2 §2A with issue text and a deploy; Viewer semantics §3.2; §7.6 step 3 and ldap_mapping.md §1 rewritten.
B2 Calling the library validator would stop OtOpcUa booting on DEV/QA (ServiceAccountDn required) Applied. §2.2 item 4 ports only the FallbackServers and BindMode/UserBindTemplate rules, with a DEV-shaped boot test. Open question 1 answered (§2.1 item 4).
M1 No logger passed to the library service Applied. §2.2 item 3.
M2 Lockout on the shared account; §1.3 had it backwards Applied. §1.3 table; D1 no-lockout ask; §7.1 value verification; HG as canary; §7.6 rule 6 stop rule.
M3 DEV automation breaks when DEV login turns on Applied. D3 covers DEV (dev-user path, ManagementAuthenticator.cs:63-71); a precondition in §7.4/§7.5.
M4 "Data plane unaffected" was misleading; pairs restarted together are an outage Applied. §7 restart rules: rolling, standby first, for config-only changes. The OtOpcUa rolling allowance is [I] pending Phase 0 / open question 5.
M5 HG group match is case-sensitive after binding Applied. §2B (optional Phase A3) with issue text; exact names from LdapProbe in §4 and D7.
M6 mxgw config-bound certificate pin Applied. Phase 0 step 4; §7.3 step 3; D4 corrected.
m §2.4 / §7.0 cross-references Applied (→ §7.5).
m Unresolved ${secret:} fails at boot Applied (§7.1, §7.6).
m Rotation needs restarts Applied (D1, §7.1).
m AcceptDomainQualifiedNames only applies in User mode Applied (§1.3; users type the bare sAMAccountName).
m D7 command was wrong Applied (-u with --svc, or --self-bind -u).
m Secrets CLI hardcoded File Applied (Phase 0 step 6 supplies the values; default noted).
m OtOpcUa secret replication Applied (Phase 0 step 6; §7.1 stores).
m mxgw vd03 can't be Production with DisableLogin Applied (§1.2; Phase 0 step 4; §7.3 step 4).
m mxgw base GwAdmin/GwReader merge Applied (§4.2 delete from base; A4 checks the effective config).
m Existing OtOpcUa sessions survive cutover Applied (§7.5 step 5; ScadaBridge self-ends via refresh).
m Rollback gaps Applied (§7.7: scripts, AD lockout, rolling on QA).
m AllowInsecure accepts any certificate Applied (D4(b)).
m Stale OtOpcUa validator comment Applied (§2.1 item 5, §2.2 item 7).
m ScadaBridge SQL bypasses audit and cache Applied (§4.4: CLI/UI preferred; SQL fallback with a 30 s wait).
n LdapOptions doc default 10.100.0.35 vs code localhost Applied (§2.1 item 5).
n Reflection test: compare lists by content, both directions Applied (§2.2 item 5).
n §4.4 DEPLOY row conditional Applied.