Files
scadaproj/docs/plans/2026-11-15-nam-ldap-cutover.md
T

981 lines
66 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# NAM AD LDAP cutover — DEV vd03, then QA vq03/vq04 (DEV + QA EXECUTED 2026-11-19 on the interim account)
> **Status lives in Gitea.** Execution is tracked by
> [scadaproj#6](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/6). The dedicated, shared LDAP-lookup service
> account is [scadaproj#58](https://gitea.dohertylan.com/dohertj2/scadaproj/issues/58), a QA prerequisite (§5, D1). This
> file is the design and runbook. It **supersedes** [`2026-08-13-vd03-nam-ldap-cutover.md`](2026-08-13-vd03-nam-ldap-cutover.md)
> (§10 lists what changed). The independent review of 2026-11-15 is applied (§11).
**Written 2026-11-15. Status: PLAN ONLY. Nothing here has been executed.**
**EXECUTED ON DEV 2026-11-19** (§0): all four apps on vd03 log in against NAM AD. Record:
[`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md). The DC-failover checks
(§7.6 step 5, A2) are dropped by owner instruction. New QA prerequisite from DEV: OtOpcUa's `RequireHttpsCookie`
on a plain-HTTP AdminUI (record, finding 1). **QA plan: §0Q** (written 2026-11-19, not executed; QA hold stands).
**Updated 2026-11-19:** §0 is the **DEV vd03 execution plan using the groups and accounts that exist today**, so
it does not wait for scadaproj#58, the D3 automation account or the D6 deploy group. QA keeps the full
prerequisites in §5–§7, and the QA deploy hold still stands.
---
## 0. DEV vd03 now, with today's groups and accounts (2026-11-19)
### 0.1 What the directory holds today
Re-enumerated 2026-11-19 from the Mac (`ldapsearch` over LDAPS to nam1, bound as `svc_igntadmin`; that bind is
the D1 interim's single verification bind).
- **Groups:** 41 `UG_WW_*`, the 39 from 08-12 plus `UG_WW_PAR_CVD_Oper` and `UG_WW_PAR_CVD_Super`. Neither is a
platform group, so neither is mapped. **There is still no `UG_WW_APP_DEPLOY_*` group.**
- **Names:** the DEV platform names are exactly as in §4. Copy them with this case.
| DEV group | Members (2026-11-19) |
|---|---|
| `UG_WW_APP_ADMIN_DEV` (7) | chamalas, millerh5, benjamd2, marting4, dohertj2, oswaltk, Luis Bastidas |
| `UG_WW_APP_IDE_DEV` (6) | chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
| `UG_WW_APP_IDE_RO_DEV` (2) | benjamd2, oswaltk |
| `UG_WW_HIST_ADMIN_DEV` (7) | **the same 7 people as `APP_ADMIN_DEV`** |
| `UG_WW_CITRIX_DEV` (6) | never mapped; culverj holds only this group, so they get no role on DEV |
**Accounts used on DEV (no new accounts):**
| Use | Account | Groups and resulting roles | Credential |
|---|---|---|---|
| LDAP lookup (D1 interim) | `svc_igntadmin@nam.zimmer.com` | No `UG_WW_*` group, so it **cannot sign in** to any app, which is correct for a lookup account | Mac Keychain, service `btdb-manager/ldap`. Seeded into each store as `nam-ldap-svc-password` |
| Positive login and automation (D3 option b) | `dohertj2` | `APP_ADMIN_DEV` + `APP_IDE_DEV` + `HIST_ADMIN_DEV`. Gets Administrator everywhere, plus Designer in OtOpcUa and ScadaBridge, plus Deployer in ScadaBridge (§0.2) | Mac Keychain (`NAM\dohertj2`) |
| Negative control: zero roles | `adm_dohertj2` | 3 groups, none `UG_WW_*`. The library passes it (`memberOf` is non-empty), the app maps no role, and §3.2 must refuse it | Mac Keychain (`NAM\adm_dohertj2`) |
| Negative control: bad credentials | a bogus user/password | — | — |
**Viewer is not exercised live on DEV.** Only benjamd2 and oswaltk hold `APP_IDE_RO_DEV`, and both also hold
`APP_ADMIN_DEV`. The Viewer mapping is checked by reading the effective config back (A4). A real Viewer-only
login waits for QA or for a volunteer.
### 0.2 Deviations from the full plan (DEV only)
| Full plan | DEV now | Why it's acceptable | Ends when |
|---|---|---|---|
| D1: dedicated account from scadaproj#58 | `svc_igntadmin` | The interim §5 already allows on DEV. **Tell the Ignition team before §0.4 step 3**: a wrong secret here locks *their* admin account (2 bad binds per login, §1.3) | scadaproj#58 is delivered; swap it in during the soak (§7) |
| D3: a NAM automation account | `dohertj2` (option b) | It holds every DEV role the scripts need | The D3 account exists |
| D6: `UG_WW_APP_DEPLOY_DEV` → Deployer | **`UG_WW_HIST_ADMIN_DEV` → `Deployer`, in ScadaBridge only** | Still one group, one role per app (§3.1): ScadaBridge maps `HIST_ADMIN` to nothing else. Its members are identical to `APP_ADMIN_DEV`, so the effect is "DEV admins can deploy", the SoD collapse already accepted for this directory (`ldap_setup.md`). It is a DEV-only row | D6's group exists: add `APP_DEPLOY_DEV` → Deployer, then delete the `HIST_ADMIN_DEV` row |
| D4: `zimmer-rootca` in `LocalMachine\Root` | Phase 0 checks it. vd03 is domain-joined, so the root is expected to be there via GPO [I]. If it isn't, run DEV with `AllowInsecure=true` and record it | DEV only, short-lived (D4 b) | The root is installed |
| D7: membership confirmed per person | Done for DEV by §0.1 | — | — |
**Unchanged:** the standards (§3), the LDAP block (§6, with `ServiceAccountDn: svc_igntadmin@nam.zimmer.com`),
the restart rules, the §7.6 verification and stop rule, and the rollback (§7.7).
### 0.3 Builds on vd03
| App | On vd03 (2026-11-19) | Carries | Action |
|---|---|---|---|
| HistorianGateway | `f84af29` | #136 (case-insensitive `GroupToRole`, mapper fault refused) | none |
| MxAccessGateway | `e74918cc` | already refuses zero roles | none |
| OtOpcUa | `cdf433a35` | #913 (§2): options passthrough, zero-role refusal, library logger | none |
| ScadaBridge | **`061432c8`** | **neither #144 nor #145** | **Deploy `main` `2e0ce81e`** (§0.4 step 1) |
ScadaBridge `061432c8..2e0ce81e` is 11 commits: #144, #145 and the #116 `leader-auto-down` default (already
drilled on vd03, 2026-11-15). **No migration** and no wire change. Use the procedure in
`docs/deployment/deployment-records/2026-11-15-wonder-app-vd03-061432c8.md` (single central + site; take a
DB backup first). With login still off, the dev user holds every role, so #144/#145 have no visible effect
until step 6.
### 0.4 Order of work on vd03
All remote steps stay inside the `winrm` deploy scope (§7). One app at a time; each one passes §7.6 before the next.
1. **ScadaBridge binary deploy `2e0ce81e`.** Verify with the deploy's normal checks: health, the site connected,
the CLI works as the login-off dev user.
2. **Phase 0 on vd03 (§7.0), reduced:** DC reach on 636 from vd03; the `zimmer-rootca` check (decides D4);
back up every app's config and env; record OtOpcUa `Security__Auth__DisableLogin` (expected `true`), mxgw's
environment name, its `ServerCertificateThumbprintSha256` and base `GwAdmin`/`GwReader`; each app's secrets
store and KEK source; the OtOpcUa `LdapGroupRoleMapping` rows. Builds per §0.3. The service-account bind was
done 2026-11-19 (§0.1), so don't repeat it.
3. **Seed `nam-ldap-svc-password`** in each store (§7.1): HG, mxgw and ScadaBridge Central through their
`/admin/secrets` pages (login is still off on all three), and OtOpcUa's store with the `secret` CLI or its
page. Verify each read-back value with **one** LdapProbe bind as `svc_igntadmin`.
4. **HistorianGateway (the canary), §7.2.** §6 + §4.1 DEV mapping, `Dashboard:DisableLogin=false`. Run §7.6 fully,
including the nam1→nam2 failover check.
5. **MxAccessGateway, §7.3.** §6 + §4.2 DEV mapping; delete the base `GwAdmin`/`GwReader`; remove any GLAuth pin;
`DisableLogin=false`.
6. **ScadaBridge, §7.4.**
- a. **While login is still off**, add the role mappings with the CLI (`--url` the vd03 central):
`UG_WW_APP_ADMIN_DEV`→Administrator, `UG_WW_APP_IDE_DEV`→Designer, `UG_WW_APP_IDE_RO_DEV`→Viewer,
**`UG_WW_HIST_ADMIN_DEV`→Deployer (DEV interim, §0.2)**. The §4.4 SQL fallback works too: put the
`HIST_ADMIN_DEV` row in its `@t`.
- b. Write `ScadaBridge:Security:Ldap` (§6). Remove **both** `DisableLogin` and `AllowOutsideDevelopment`.
- c. Restart the central. Re-point scripts (`apply.sh`, drill and deploy packs) at `dohertj2`, and confirm
`/auth/token` and the CLI (HTTP Basic) work with it.
- d. Extra §7.6 check: `adm_dohertj2` is refused at `/auth/login`, at `/auth/token` **and** on the
management/CLI path (#145).
7. **OtOpcUa AdminUI, §7.5.** §6 as `Security__Ldap__*` with `FallbackServers__0` (the #913 build honours it),
the §4.3 DEV `GroupToRole`, then `Security__Auth__DisableLogin=false`. Restart the single node. §7.6 plus
acceptance A2 (nam1 blocked → login through nam2, with the library's `sticky preference moved` line).
`adm_dohertj2` is refused at `/auth/login` and `/auth/token`.
8. **Record** the run as `docs/deploy/2026-11-<dd>-vd03-nam-ldap-cutover.md` in scadaproj, plus a line in each
app's deployment notes. The mxgw notes must say "delete `GwAdmin`/`GwReader` from the base file on every
redeploy". Tick the DEV items on scadaproj#6.
**Stop rule (§7.6 rule 6) for this run:** the account is Ignition's, so stop at the **first** `service-bind`
warning, not the second. Ask IT for the account's lockout state before any retry.
**Soak:** a few working days of real logins on DEV.
---
## 0Q. QA vq03/vq04, with today's groups and accounts (plan 2026-11-19; EXECUTED 2026-11-19)
**Executed 2026-11-19 15:29–16:10 EST:** all four apps on both boxes on the DEV builds and NAM AD; record
[`docs/deploy/2026-11-19-qa-nam-ldap/`](../deploy/2026-11-19-qa-nam-ldap/README.md). Optional step 8 (oswaltk) not run.
**Owner decisions (2026-11-19):**
1. QA binds as **`svc_igntadmin`** as well (interim). This supersedes A6 and the §7 note "QA never runs on
`svc_igntadmin`". Both tiers move to the scadaproj#58 account together when it arrives: re-seed
`nam-ldap-svc-password` in every store, then a rolling restart per app.
2. **QA gets exactly the builds DEV runs.**
3. **One window:** each app's code deploy and its LDAP change share one restart.
**Gate:** the owner lifts the QA hold and names the window. **Tell the Ignition team before the window:** QA adds
8 more stores that reference their admin account.
### 0Q.1 What QA holds today
Read-only checks on 2026-11-19 (directory as in §0.1; QA config files over SMB; health and redirects over HTTP).
| DEV-proven groups | QA members (2026-11-19) |
|---|---|
| `UG_WW_APP_ADMIN_QA` (6) | chamalas, millerh5, benjamd2, marting4, dohertj2, Luis Bastidas |
| `UG_WW_APP_IDE_QA` (6) | the same 6 |
| `UG_WW_APP_IDE_RO_QA` (1) | oswaltk (**Viewer only**, so a live Viewer test is possible with them) |
| `UG_WW_HIST_ADMIN_QA` (6) | **the same 6 as `APP_ADMIN_QA`**, so the §0.2 Deployer interim carries over |
| `UG_WW_BTMGR_RO_QA` (0) | empty; mapped per §4 anyway (grants nobody anything) |
**Every app already has login ON on both boxes, against the box-local GLAuth, over plain HTTP.** Anonymous `GET /`
returns 302 → `/login` on `:5220`, `:5130`, `:8085` and `:9000`. So QA is a **directory swap**, not a "turn login on". The
GLAuth `multi-role` automation identity stops working at the cutover; the D3 interim is `dohertj2`, as on DEV.
| App | QA build now | → target (= DEV) | Where its LDAP config lives (QA) | Notes |
|---|---|---|---|---|
| HistorianGateway (per box) | `6365f34` | `f84af29` (#136) | **NSSM env** `Ldap__*`, `Dashboard__GroupToRole__SCADA-*` [I] | `app\appsettings.json` has `Dashboard:RequireHttpsCookie=true`, yet login runs over HTTP. Phase 0 reads whether the env overrides it (finding 1 below) |
| MxAccessGateway (per box) | `76a71bc8` | `e74918cc` | `app\appsettings.json`: GLAuth `localhost:3893`, `dc=zb,dc=local`, `${secret:ldap/mxgateway/bind}`, **base `GwAdmin`/`GwReader`** | Phase 0 records the environment name and any overlay. Delete `GwAdmin`/`GwReader` from the base file (§4.2) |
| ScadaBridge (pair) | `061432c8` | `2e0ce81e` (#144/#145) | `app\appsettings.Central.json` (GLAuth `127.0.0.1:3893`, `dc=scadabridge`) + Central env password | **`Secrets:Store:Provider=SqlServer`**: one shared store, so seed once. `RequireHttpsCookie=false` already. The two Centrals' files differ (known per-node JWT/idle drift): edit each file, never copy one over the other |
| OtOpcUa (Warm pair) | `76e73e7d` | `cdf433a3` (#913; 83 commits, **no migration**) | service env [I] | Warm redundancy (`Mode=Warm`); Phase 0 records which node is Primary. Needs the cookie decision (finding 1) |
Secrets master keys: the shipped files say `Source=Environment` (`ZB_SECRETS_MASTER_KEY`) for HG, mxgw and OtOpcUa. The
service env may override that. The seed script reads each service's own env, so it never assumes.
### 0Q.2 Findings carried from DEV
1. **Plain-HTTP cookies.** OtOpcUa's auth cookie is `Secure` by default, so its AdminUI login loops over HTTP. On DEV
the fix was `Security__Cookie__RequireHttpsCookie=false`. QA OtOpcUa login works today over HTTP, so Phase 0 checks
whether QA already sets that. If not, set it in the cutover (same value as DEV). Check HG `Dashboard:RequireHttpsCookie`
the same way.
2. **Secrets CLI.** The boxes' CLI (0.6.2) takes a value only on argv. Use the 0.6.6 build from DEV (stdin; no store
schema or envelope change), copied to `E:\ApiInstall\_tools\secret-0.6.6\` on each box.
3. **Do not** add firewall rules or run DC-failover drills (owner, 2026-11-19).
4. Logs: the apps' own files may be empty. Verification reads the box OTel collector export
(`E:\ApiInstall\OtelCollector\data\logs.jsonl`) over SMB.
### 0Q.3 Mappings (QA suffix)
As §4, plus the Deployer interim:
| Group | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
| `UG_WW_APP_ADMIN_QA` | Administrator | Administrator | Administrator | Administrator |
| `UG_WW_APP_IDE_QA` | Designer | Designer | Viewer | — |
| `UG_WW_APP_IDE_RO_QA` | Viewer | Viewer | Viewer | Viewer |
| `UG_WW_HIST_ADMIN_QA` | Viewer | **Deployer (interim)** | — | Administrator |
| `UG_WW_BTMGR_RO_QA` | — | Viewer | — | Viewer |
Remove every GLAuth mapping: HG `SCADA-*` env entries and mxgw `GwAdmin`/`GwReader`. The ScadaBridge `SCADA-*` seed rows
stay; they grant nothing in NAM.
### 0Q.4 The window
All remote steps are `winrm` deploy scope on vq03/vq04, one app at a time, in the canary order. **Stop at the first
`service-bind` warning** (§0.4 stop rule). Each app's code deploy and LDAP change share one restart.
0. **Before the window (from the Mac, no remote exec):**
- stage the four builds (same artifacts as DEV, re-hashed) and the 0.6.6 Secrets CLI over SMB;
- **add the ScadaBridge role rows with the CLI** while QA still logs in through GLAuth. Log in as `multi-role`,
with `--url` pointing at the **active** central. This path is audited and invalidates the cache, unlike the SQL path DEV
had to use. Add the five rows from §0Q.3; they grant nothing until the cutover.
1. **Phase 0 per box** (`preflight.ps1` from the DEV record, retargeted to the QA layout `E:\ApiInstall\<App>\app`
and the service names):
- DC reach on 636 and `zimmer-rootca` trust;
- back up every service key and every `appsettings*.json` to `E:\ApiInstall\_ldapcut-<ts>\`;
- record each app's LDAP/login/cookie env, mxgw's environment name, OtOpcUa `Secrets:Replication:Enabled`, the
OtOpcUa Primary/Standby and ServiceLevels, and the `LdapGroupRoleMapping` overlay rows;
- read each app's secrets provider and master-key source from its service env.
2. **Seed `nam-ldap-svc-password`** (`seed-secret.ps1`, generalised to read each service's own `Secrets__*` and
master-key env):
- HG and mxgw: per box;
- OtOpcUa: per box, or once if replication is on;
- ScadaBridge: **once**, into the shared SQL store, using the active Central's provider settings.
Read back each value and compare its hash against the Keychain value. The value file is ACL'd and deleted.
3. **HistorianGateway**, vq03 then vq04 (independent single nodes; this is the canary):
- deploy `f84af29` with HG's own procedure (`HistorianGateway/docs/deploy/2026-11-19-vd03-f84af29/` retargeted to
QA);
- before the start, rewrite the NSSM env: `Ldap__*` → §6, `Dashboard__GroupToRole__*` → §0Q.3, and the cookie per
finding 1.
Verify on each box: bogus refused, `dohertj2` → Administrator, no `system-side` lines. Then the next box.
4. **MxAccessGateway**, vq03 then vq04:
- deploy `e74918cc` with mxgw's procedure. It will replace `app\appsettings.json`, so apply the edit to the **new**
file before the start: §6 block, QA `GroupToRole`, delete `GwAdmin`/`GwReader`, no GLAuth pin;
- mind the nssm `START_PENDING` abort in the repo upgrade script (memory note: use the try/catch fix as on vd03).
Verify per box as in step 3.
5. **ScadaBridge pair**: deploy `2e0ce81e` with the QA pair procedure (`deploy-prep/2026-11-15-143/qa.tmpl.ps1` actions
minus `migrate`):
- stop both Sites, then both Centrals, so the pair goes down together;
- swap the binaries and edit **each** Central's `appsettings.Central.json`: §6 block, drop any `Auth` block;
- remove `ScadaBridge__Security__Ldap__ServiceAccountPassword` from **both** Central envs;
- start the Centrals (vq03 first), then the Sites.
Verify:
- `dohertj2` gets Administrator+Designer+Deployer at `/auth/login`, at `/auth/token` and on the CLI against the
active central;
- bogus refused;
- LocalDb sync connected; Z28062–Z28069 at their Good baseline; `WarmCanary` connected; one active per pair.
6. **OtOpcUa Warm pair**: deploy `cdf433a3` with OtOpcUa's QA procedure (the 2026-11-13 `76e73e7d` pack: stop-both /
start-both for a binary upgrade, worker isolation `-Reapply`/`-Verify`).
- **Before the start**, add to **both** nodes' env: the 18 `Security__Ldap__*` entries from the DEV record (QA suffix
in `GroupToRole`), `Security__Auth__DisableLogin` left `false`, and the cookie per finding 1.
- Verify:
- ServiceLevel 250 on the Primary, 90 on the Standby (Warm), and the Standby canary reads `BadOutOfService`;
- LocalDb sync connected; the script-runner canary Good;
- `dohertj2` → Viewer+Designer+Administrator via cookie and `/auth/token`; bogus refused;
- the ScadaBridge `WarmCanary` reconnects.
7. **Re-point QA automation** from `multi-role` to `dohertj2`: the qa pack scripts, drills, `apply.sh`.
8. **Optional:** oswaltk's Viewer-only login (the only Viewer-only member).
9. **Record** in `docs/deploy/2026-11-<dd>-qa-nam-ldap/`, plus a deploy record in each repo.
**Outages:** HG and mxgw a few seconds per box. The ScadaBridge pair stops together (about 2–3 min of site data path on
the 11-15 run). The OtOpcUa pair stops together (about 37 s on 11-13). That is the price of one window: every restart is
also a binary upgrade, and the pairs can't roll those.
**Rollback:** per app, restore the `_ldapcut-<ts>` keys and files. Binaries roll back with each repo's own procedure
(no app here has a schema change). Keep the ScadaBridge `UG_WW_*` rows. The box-local GLAuth stays running until the
soak ends.
**Goal:** move the login of the four auth-bearing apps off their current setup (disabled login, or a GLAuth
test directory) and onto the Zimmer NAM Active Directory: DEV first, a soak, then the QA pair.
- The service-account password lives as a `${secret:}` reference in each app's encrypted Secrets store, never
in config.
- Every app maps AD groups to its roles from one canonical matrix, [`ldap_mapping.md`](../../ldap_mapping.md),
under two family rules (§3):
- **one AD group grants exactly one role per app;**
- **a user whose groups map to no role cannot sign in** (cookie or token).
**In scope:** HistorianGateway (dashboard), MxAccessGateway (dashboard), ScadaBridge (Central UI + CLI),
OtOpcUa (AdminUI).
**Out of scope:**
- **Overview:** no login by design.
- **The machine-to-machine surfaces stay API-key based:** OtOpcUa's OPC UA data plane, the mxgw and HG gRPC
APIs, and ScadaBridge's inbound API. They can still be *disrupted* by the restarts here (§7.0).
- **ignitionoee:** on hold (`ldap_mapping.md` §6).
**Evidence key:** **[V]** = verified in source or a file, with the citation; **[I]** = inferred, to be confirmed
in Phase 0. Source paths are relative to each repo.
---
## 1. Current state (2026-11-15)
### 1.1 Builds and the Auth library
The 2026-08-13 plan's "redeploy to Auth 0.2.1" phase is **obsolete**:
| App | Deployed on DEV and QA | Auth pin [V] |
|---|---|---|
| OtOpcUa | `76e73e7d` | **0.4.1** (`Directory.Packages.props:238-240`) |
| ScadaBridge | `061432c8` | **0.5.0** (ScadaBridge#46) |
| MxAccessGateway | `76a71bc8` | **0.4.1** |
| HistorianGateway | `6365f34` | **0.4.1** |
Domain-root `SearchBase` (≥ 0.2.1), `FallbackServers` (≥ 0.2.0 in the library, but **dropped by OtOpcUa**,
§2) and `BindMode` with the empty-password guard (≥ 0.4.1) are all available.
**Before the cutover:**
- **OtOpcUa (§2)** and **ScadaBridge (§2A)** need code changes and a deploy.
- **HistorianGateway (§2B)** gets an optional hardening change.
### 1.2 Login state per app and tier
| App | DEV vd03 | QA vq03/vq04 |
|---|---|---|
| **HistorianGateway** | **Login OFF**: `Dashboard:DisableLogin=true` in `appsettings.Production.json`. LDAP points at the shared GLAuth `10.100.0.35:3893`, plain [V, file]. | **Login ON** against the box-local GLAuth. Config is in the **NSSM env** (`Ldap__*`, GroupToRole `SCADA-Admins`/`SCADA-Designers`) [I]. KEK at `C:\ProgramData\ZB\kek\historiangw.kek` [V, 11-13 evidence]. |
| **MxAccessGateway** | **Login OFF**: `MxGateway:Dashboard:DisableLogin=true` [V, file]. LDAP `127.0.0.1:3893`, `dc=scadalink,dc=local`. The deployed build **refuses** `DisableLogin=true` under `Production` (`GatewayOptionsValidator.cs:484-488`), so vd03 mxgw is **not** running as Production [V by inference from it booting]; Phase 0 records the environment name. | **Login ON**: `localhost:3893`, `dc=zb,dc=local`, `${secret:ldap/mxgateway/bind}` [V, file]. ⚠ `deployment.md` says the box-local GLAuth is `dc=scadabridge,dc=local` [I]. |
| **ScadaBridge** | **Login OFF**: `ScadaBridge:Security:Auth:DisableLogin=true` + `AllowOutsideDevelopment=true` [V, file]. With login off, the management/CLI path authenticates every caller as the all-roles dev user (`ManagementService/ManagementAuthenticator.cs:63-71`: `"multi-role"`, `Roles.All`) [V]. LDAP `localhost:3893`. | **Login ON**, box-local GLAuth `127.0.0.1:3893` [V]. Every CLI, deploy and drill script logs in as GLAuth `multi-role`. |
| **OtOpcUa** (AdminUI) | Code defaults (`Enabled=true`, `Server=localhost`, `Port=3893`, `SearchBase=dc=zb,dc=local`, `ServiceAccountDn=""`), plus files holding `Transport=Ldaps` and `DevStubMode=false`; no `Security__Ldap__*` in the env [V, 11-13 record]. `Security__Auth__DisableLogin` is in the env, **value unknown**. | Unknown; Phase 0 reads it. |
### 1.3 Directory facts
**Still valid, from `ldap_setup.md`:**
- **DCs:** `uswap-vdc-nam1.nam.zimmer.com` (`10.201.216.2`) and `uswap-vdc-nam2.nam.zimmer.com` (`10.201.138.78`).
- **Transports:** LDAPS 636, StartTLS and plain 389 all work.
- **TLS:** the certs are RSA-1024, chain `leaf ← zimmer-issueca1 ← zimmer-rootca`. The only chain error is
`UntrustedRoot`, so installing the root is sufficient. Windows accepts the 1024-bit key.
- **Directory shape:** `SearchBase` `DC=nam,DC=zimmer,DC=com`; login attribute `sAMAccountName`; `memberOf`
collapses to bare CNs.
- **Reachability:** from the Mac, nam1/nam2 on 389 and 636 are open (2026-11-15). Not proven from the boxes.
**Login names.** In `BindMode=ServiceAccount` (this plan), `AcceptDomainQualifiedNames` has **no effect**: it
applies only in User mode (`ZB.MOM.WW.Auth.Ldap/LdapAuthService.cs:156-161`) [V]. So `NAM\jdoe` and
`jdoe@nam.zimmer.com` return *user not found*. **Users must type their bare sAMAccountName.** Put that on the
login-page notice.
**Lockout (AD policy: 5 failed binds per 15 min, per account) [V, `LdapAuthService.cs`]:**
| Failure | Bad binds | Effect |
|---|---|---|
| **Wrong *user* password** | **1**, on one DC (`:164-168`, `:428-436`) | Final (`BadCredentials`); no walk to the next DC. Locks only that user, after 5 tries. |
| **Wrong *service-account* password** | **2 per login attempt**: system-side, so it walks to the next DC (`:173-174`, `:408-415`) | Locks the **shared** account. |
The shared account is one account for all four apps and both tiers (D1). A mistyped secret in any of the ~7
stores can therefore lock out login everywhere within a few attempts.
ScadaBridge's background group refresh also binds as this account. Its throttle is 5 per 5 min
(`ScadaBridge Security/SecurityOptions.cs:132-145`) [V], looser than AD's 5 per 15 min. **Mitigations:**
- D1 asks IT for a no-lockout (or very high threshold) policy on the account.
- §7.1 verifies every store's value before that app restarts.
- §7 does one canary app first.
- §7.6 has a stop rule.
---
## 2. Phase A — OtOpcUa code fix (prerequisite for the OtOpcUa cutover)
One OtOpcUa change, one issue, one deploy. It fixes five things.
### 2.1 Defects [V]
1. **Options dropped.** `LdapOptions.ToLibraryOptions()` (`src/Server/ZB.MOM.WW.OtOpcUa.Security/Ldap/LdapOptions.cs:111`)
copies 12 fields into the library's `LdapOptions` (`ZB.MOM.WW.Auth.Abstractions/Ldap/LdapContracts.cs:21`).
- **Dropped:** `FallbackServers` (0.2.0), `BindMode`, `UserBindTemplate` and `AcceptDomainQualifiedNames`
(0.4.0). The config keys are ignored without error, so the AdminUI has **no DC failover**.
- **Not config-bound, by design:** `ServerCertificateValidationCallback`, which is code-only.
2. **Zero-role sign-in (review B1).** `Security/Endpoints/AuthEndpoints.cs:124-158` signs in with an **empty**
role set when the mapper returns no roles, **and** when the mapper throws: it catches the exception, logs a
warning and continues with the baseline roles, which are empty on the real path.
- The fallback policy and `AdminUiPolicies.AuthenticatedRead` (`Security/ServiceCollectionExtensions.cs:192,217`)
only require an authenticated user, and 18 AdminUI pages use `AuthenticatedRead`.
- No policy requires `Viewer`; it is only a default in `RoleGrants.razor:126`.
- With a domain-root `SearchBase`, **any NAM user with any group would get AdminUI read access**, and
`/auth/token` would mint them a token.
3. **No library logger (review M1).** `OtOpcUaLdapAuthService.cs:48` builds `LibLdapAuthService` with no
logger, and the library logs only via `_logger?.`. So the failover log line
(`sticky preference moved`) and the stage diagnostics (`failed system-side at …`) **never appear** in
OtOpcUa, and §7.6's checks can't be done.
4. **No validation of the new options (review B2).** OtOpcUa **never runs the library's
`LdapOptionsValidator`**: it constructs the library service directly (`OtOpcUaLdapAuthService.cs:48`). That
answers the old open question.
- Calling the library validator would **stop OtOpcUa booting on DEV and QA**. It requires
`ServiceAccountDn` in ServiceAccount mode (`ZB.MOM.WW.Auth.Ldap/LdapOptionsValidator.cs:81-87`), DEV runs on
code defaults (`Enabled=true`, `ServiceAccountDn=""`), and OtOpcUa validates LDAP on every node
(`Host/Program.cs:190-195`).
5. **Stale docs.** The OtOpcUa validator comment (`Host/Configuration/LdapOptionsValidator.cs:12-14`) describes a
"direct-bind path" that doesn't exist. The `LdapOptions` doc says the default server is `10.100.0.35`; the
code says `localhost`.
### 2.2 The fix
1. **Add the four properties** with the library's defaults and pass them through `ToLibraryOptions()`:
- `List<string> FallbackServers = []` → `IReadOnlyList<string>`
- `LdapBindMode BindMode = ServiceAccount`
- `string UserBindTemplate = ""`
- `bool AcceptDomainQualifiedNames = true`
2. **Refuse a zero-role login (the family standard, §3.2).** In `/auth/login` **and** `/auth/token`:
- if the mapped role set is empty, refuse exactly like bad credentials (HTTP 401 / the login page's
"invalid or not authorized"), with no cookie and no token;
- if the mapper **throws**, also **refuse** (fail closed), log the exception, and do not sign in.
- **This reverses the current code comment's intent and the test
`Login_when_role_mapper_throws_signs_in_with_no_role_claims`**
(`tests/Server/ZB.MOM.WW.OtOpcUa.Security.Tests/AuthEndpointsIntegrationTests.cs:402`). Rewrite that test to
assert refusal.
- Add tests: empty mapping refused (cookie + token); mapper throws refused (cookie + token); one mapped role
signs in.
- **DevStub is unaffected:** it grants `Administrator`.
3. **Inject the logger:** pass `ILogger<ZB.MOM.WW.Auth.Ldap.LdapAuthService>` from DI into the
`LibLdapAuthService` constructor at `OtOpcUaLdapAuthService.cs:48`. Add a test that a failover writes the
library's `sticky preference moved` line.
4. **Validation, OtOpcUa's own validator only.** In `Host/Configuration/LdapOptionsValidator.cs`, port **only**:
- the library's `FallbackServers` rule (each entry parses as `host[:port]`);
- the `BindMode=User` → `UserBindTemplate` contains `{0}` exactly once rule.
**Do not** port the `ServiceAccountDn`-required rule.
**Add a test that boots the validator with today's DEV-shaped config** (`Enabled=true`, `Server=localhost`,
`ServiceAccountDn=""`, `Transport=Ldaps`) and asserts it **passes**.
5. **Completeness guard.** Add a unit test that reflects over the library `LdapOptions`' public settable
properties.
- For each one, set a non-default value on OtOpcUa's options, call `ToLibraryOptions()`, and assert the value
arrived. Compare `IReadOnlyList` values **by content**.
- Check **both directions:** every library property is mapped or allow-listed, and no allow-list entry is
stale.
- Allow-list: `ServerCertificateValidationCallback`.
- Mutation check: delete one assignment and the test must fail.
6. **Binding test:** `Security:Ldap:FallbackServers:0` reaches the library options, in JSON and as the env var
`Security__Ldap__FallbackServers__0`.
7. **Docs:**
- `docs/Configuration.md` `Security:Ldap`: the new keys, a NAM two-DC example, the zero-role refusal, and
"type the bare sAMAccountName".
- Fix the two stale comments from §2.1 item 5.
8. **Acceptance A1 (offline):** an AdminUI login against the redundant GLAuth pair (`infra/glauth-redundant/`,
`:3893` + `FallbackServers ["…:3894"]`).
- Stop the primary: login still succeeds and the log shows `sticky preference moved to …:3894`.
- A GLAuth user in no mapped group is refused.
### 2.3 Issue text for `lmxopcua` (not filed by this plan)
> **Title:** AdminUI LDAP: drops `FallbackServers`/`BindMode`/…, signs in users with no mapped role, and the library logs nothing
>
> **Labels:** `type:bug`, `priority:p1`, `area:auth`, `cross-repo`
>
> **Body:**
> 1. `LdapOptions.ToLibraryOptions()` (`Security/Ldap/LdapOptions.cs:111`) drops `FallbackServers`, `BindMode`,
> `UserBindTemplate` and `AcceptDomainQualifiedNames`, so there is no AD DC failover.
> 2. `AuthEndpoints.cs:124-158` signs in with an empty role set, including when the mapper throws, and
> `AuthenticatedRead` admits any authenticated user. Once the search base is a whole AD domain, any directory
> user gets AdminUI read access and a token. The fix refuses empty roles at `/auth/login` and `/auth/token`,
> and refuses when the mapper throws. This reverses `Login_when_role_mapper_throws_signs_in_with_no_role_claims`.
> 3. `OtOpcUaLdapAuthService.cs:48` passes no logger to the library, so failover and stage logs are missing.
> Inject `ILogger<LdapAuthService>`.
> 4. Validate the new options in OtOpcUa's own validator: the fallback parse rule and the BindMode/UserBindTemplate
> rules only. **Not** `ServiceAccountDn`, because DEV boots on code defaults with an empty DN.
> 5. A two-way reflection completeness test, a binding test, docs, and fixes to the stale comments.
>
> **Acceptance:** failover on the redundant GLAuth pair, a zero-role user refused (cookie + token), and a
> DEV-shaped config boots. Plan: `scadaproj/docs/plans/2026-11-15-nam-ldap-cutover.md` §2.
### 2.4 Ship it
Merge to `master` and deploy with OtOpcUa's own procedure (`docs/deploy/`, most recently
`2026-11-13-vd03-qa-to-76e73e7d.md`): DEV, then QA.
- **No LDAP config change** goes out with this deploy.
- **The zero-role refusal does change behaviour on QA:** it is live against the box-local GLAuth today, so any
GLAuth user without a mapped group loses AdminUI read access. Phase 0 checks that the `multi-role` user maps.
- **Gate:** §7.5 (the OtOpcUa cutover) does not start until the deployed build contains this fix.
---
## 2A. Phase A2 — ScadaBridge zero-role refusal (prerequisite for the ScadaBridge cutover)
**Defect [V, review B1]:**
- `CentralUI/Auth/AuthEndpoints.cs:59-115` signs in (cookie) and issues a bearer at `/auth/token` (`:125`) even
when `RoleMapper` returns no roles.
- `Dashboard.razor:2` and `Monitoring/Health.razor:2` are bare `[Authorize]`, so any directory user could read
them.
**Fix:**
1. In `/auth/login` and `/auth/token`, an empty mapped role set returns the bad-credentials response, with no
cookie and no token.
2. Apply the same rule to the mid-session group refresh (`DirectoryGroupRefresher`). A session whose roles drop
to zero is already ended by that refresh (`:141-142`); keep that behaviour and test it.
3. **Tests:** zero roles refused (login + token); one role signs in; a refresh to zero ends the session.
4. **Docs:** the `Component-Security` / CentralUI auth section.
5. **Leave the bare `[Authorize]` pages** as they are; they are safe once a login requires a role. Add a
static test that every page is either policy-gated or relies on that rule.
**Issue text for `ScadaBridge` (not filed):**
> **Title:** Central UI / `/auth/token` sign in users whose AD groups map to no role
>
> **Labels:** `type:bug`, `priority:p1`, `area:auth`, `cross-repo`
>
> **Body:** `CentralUI/Auth/AuthEndpoints.cs:59-115` signs in and mints a bearer with an empty role set. With a
> domain-wide NAM search base, any AD user reaches `Dashboard` and `Monitoring/Health` (bare `[Authorize]`). The
> family standard (scadaproj NAM cutover plan §3.2) refuses login when no role is mapped. Fix login and token,
> keep the refresh's end-on-zero, and add tests.
>
> **Acceptance:** a zero-role user is refused at both endpoints. Plan: `docs/plans/2026-11-15-nam-ldap-cutover.md` §2A.
**Ship:** a ScadaBridge build containing the fix, deployed DEV then QA before §7.4. This is a binary deploy, so
it follows the QA pair rules: stop both centrals and start them together.
---
## 2B. Phase A3 — HistorianGateway case-insensitive group match (optional hardening)
**Defect [review M5]:** `Dashboard/DashboardOptions.cs:44-45` declares `IReadOnlyDictionary<string,string>`
with an `OrdinalIgnoreCase` default. The configuration binder **replaces** it with a new case-**sensitive**
`Dictionary`. The reviewer proved this with an offline binder test; the XML doc says "compared
case-insensitively", which is wrong once it's bound. AD group names are mixed case, so a key whose case differs
from the `memberOf` CN never matches.
**Fix:** declare `Dictionary<string,string>` with `StringComparer.OrdinalIgnoreCase`, so the binder fills the
existing instance, as mxgw and OtOpcUa do. Add a binder test with a mixed-case key.
**Until it ships (and in any case),** §4 copies each group name **exactly** as LdapProbe prints it.
**Issue text for `historiangw` (not filed):**
> **Title:** `Dashboard:GroupToRole` is case-sensitive after binding (docs say case-insensitive)
>
> **Labels:** `type:bug`, `priority:p3`, `area:auth`
>
> **Body:** the `IReadOnlyDictionary` default with `OrdinalIgnoreCase` is replaced by the binder's default
> `Dictionary`. Make the property a `Dictionary` with `OrdinalIgnoreCase` and add a binder test with a mixed-case
> key. Found by the NAM cutover review (scadaproj plan §2B).
---
## 3. The family standards (DECIDED 2026-11-15)
### 3.1 One AD group grants exactly one role per app
A user's roles in an app are the union of the roles of every mapped group they belong to. No app maps one group
to two roles. Each deployment maps only its own `_DEV`/`_QA`/`_PROD` suffix.
**Why:**
1. Every schema already enforces or implies it, so no code change is needed:
| App | Mapping shape [V] |
|---|---|
| OtOpcUa | `Dictionary<string,string> GroupToRole` (`Security/Ldap/LdapOptions.cs:84`), unioned with the DB overlay `LdapGroupRoleMapping` (`OtOpcUaGroupRoleMapper.cs`). The overlay's only unique index is `(LdapGroup, ClusterId)`, filtered, so two system-wide rows per group are possible; **the rule forbids them** and Phase 0 checks. |
| ScadaBridge | table **`LdapGroupMappings`**, with `HasIndex(LdapGroupName).IsUnique()` (`ConfigurationDatabase/Configurations/SecurityConfiguration.cs:26`). `RoleMapper` unions roles across different groups (`Security/RoleMapper.cs:48-60`). |
| MxAccessGateway | `Dictionary<string,string>` (`Configuration/DashboardOptions.cs:69`). The validator allows only `Administrator`/`Viewer` (`GatewayOptionsValidator.cs:506-517`). |
| HistorianGateway | `IReadOnlyDictionary<string,string>` (`Dashboard/DashboardOptions.cs:44`). Case-sensitive in practice (§2B). |
2. Grants stay readable in AD, where IT owns membership.
3. It removes a silent failure: the old two-row ScadaBridge insert hits the unique index.
**Rejected:** several roles per group. It would need a ScadaBridge migration and matching changes in the other
apps. It stays a future option.
### 3.2 No role, no login (added by review B1)
**Every app refuses sign-in, both cookie and token, when the user's groups map to no role.** That includes the
case where the role lookup fails.
| App | Before the fixes [V] | Change (status 2026-11-17: merged, **not deployed**) |
|---|---|---|
| MxAccessGateway | refuses (`Dashboard/DashboardAuthenticator.cs:66-70`) | none |
| HistorianGateway | refused zero roles; a throwing mapper escaped as a 500 | historiangw#136, `f84af29`: case-insensitive `GroupToRole`, a mapper fault is refused like a bad password, both logged at Warning (`Dashboard/DashboardAuthenticator.cs`, the `MapAsync` catch and the `roles.Count == 0` branch) |
| OtOpcUa | signed in with no roles, even when the mapper threw | **§2 (Phase A)**: lmxopcua#913, `cdf433a35` |
| ScadaBridge | signed in with no roles: cookie, `/auth/token`, management HTTP Basic/Bearer | **§2A (Phase A2)**: #144 `d48e1667` (cookie, token, session refresh) + #145 `2e0ce81e` (management/CLI, audit, debug stream) |
**Consequence for the matrix:** read access now **requires** a mapped group. In OtOpcUa, `Viewer` is not checked
by any policy (pages use `AuthenticatedRead`), so a **Viewer cell means "holds a role, so may sign in and read"**.
That is why `APP_IDE_RO_<T>` → `Viewer` stays, and why `HIST_ADMIN_<T>` → `Viewer` in OtOpcUa is meaningful.
### 3.3 Consequence for ScadaBridge `Deployer`
`APP_IDE` → `Designer` only. In ScadaBridge, `Administrator` does not imply `Deployer`: each policy requires its
own role (`Security/AuthorizationPolicies.cs:190-202`). No existing group means "deploy", so:
- a **new AD group `UG_WW_APP_DEPLOY_{DEV,QA}`** (D6) maps to `Deployer`;
- until it exists on a tier, **the ScadaBridge cutover does not run there**, because nobody could deploy.
The other apps have no `Deployer` role; OtOpcUa deploy is `ConfigEditor` = Administrator/Designer,
`AdminUiPolicies.cs:18`.
---
## 4. The tier mappings (generated from `ldap_mapping.md` §4.1)
These are platform groups only. Plant-floor, site and batch groups are not in this cutover (D5), and `CITRIX_*`
is never mapped. **Copy each group name exactly as LdapProbe prints it** (§2B; D7). The names below are as
observed on 2026-08-12.
| Group (`<T>` = `DEV` or `QA`) | OtOpcUa | ScadaBridge | MxAccessGateway | HistorianGateway |
|---|---|---|---|---|
| `UG_WW_APP_ADMIN_<T>` | Administrator | Administrator | Administrator | Administrator |
| `UG_WW_APP_IDE_<T>` | Designer | Designer | Viewer | — |
| `UG_WW_APP_DEPLOY_<T>` *(new, D6; only once it exists)* | — | Deployer | — | — |
| `UG_WW_APP_IDE_RO_<T>` | Viewer | Viewer | Viewer | Viewer |
| `UG_WW_HIST_ADMIN_<T>` | Viewer | — | — | Administrator |
| `UG_WW_BTMGR_RO_QA` *(QA only; empty as of 08-12)* | — | Viewer | — | Viewer |
### 4.1 HistorianGateway (DEV: `appsettings.Production.json`; QA: NSSM env)
```jsonc
"Ldap": { /* §6 */ },
"Dashboard": {
"DisableLogin": false,
"GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_HIST_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
}
```
On QA, remove every `Dashboard__GroupToRole__SCADA-*` entry first, then set:
```
Dashboard__GroupToRole__UG_WW_APP_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_HIST_ADMIN_QA=Administrator
Dashboard__GroupToRole__UG_WW_APP_IDE_RO_QA=Viewer
Dashboard__GroupToRole__UG_WW_BTMGR_RO_QA=Viewer
```
### 4.2 MxAccessGateway (the overlay file for the box's environment name; Phase 0 records it)
The **base `appsettings.json` ships `GwAdmin`/`GwReader` mappings** (`:86-89`) [V], and dictionaries **merge**
across overlays. So the overlay alone can't remove them, and they can't be neutralised either, because the
validator accepts only `Administrator`/`Viewer` as a value. **Delete the two entries from the box's base
`appsettings.json`** (backed up). A redeploy restores the base file, so the mxgw deploy notes must carry the
deletion forward.
Then set:
```jsonc
"MxGateway": {
"Ldap": { /* §6 */ },
"Dashboard": {
"DisableLogin": false,
"GroupToRole": { "UG_WW_APP_ADMIN_DEV": "Administrator", "UG_WW_APP_IDE_DEV": "Viewer", "UG_WW_APP_IDE_RO_DEV": "Viewer" }
}
}
```
On QA, use the `_QA` suffix and delete `GwAdmin`/`GwReader` the same way. **Acceptance A4 reads the *effective*
config** (§8), not just the edited file.
### 4.3 OtOpcUa (service env, both tiers)
```
Security__Ldap__GroupToRole__UG_WW_APP_ADMIN_<T>=Administrator
Security__Ldap__GroupToRole__UG_WW_APP_IDE_<T>=Designer
Security__Ldap__GroupToRole__UG_WW_APP_IDE_RO_<T>=Viewer
Security__Ldap__GroupToRole__UG_WW_HIST_ADMIN_<T>=Viewer
```
- **DB overlay:** the RoleGrants page / `LdapGroupRoleMapping` must hold **no** rows for these groups, or only the
same role.
- `DriverOperator` also honours the role `Operator` (`ServiceCollectionExtensions.cs:201-202`); nothing maps to
it here.
### 4.4 ScadaBridge (central database)
**Preferred: the management surface**, so the change is audited by the ManagementActor and the 30 s
cross-central role-mapping cache is invalidated.
- Use the CLI `security` role-mapping commands: `scadabridge security --help` lists them. Run them on the target
tier with an explicit `--url`, **while DEV login is still off** (the dev user holds every role).
- On QA, run them as the current GLAuth admin before the cutover.
- The Central UI's role-mapping page is equivalent.
Add one mapping per group:
- `UG_WW_APP_ADMIN_<T>` → `Administrator`
- `UG_WW_APP_IDE_<T>` → `Designer`
- `UG_WW_APP_IDE_RO_<T>` → `Viewer`
- `UG_WW_APP_DEPLOY_<T>` → `Deployer`, **only if that group exists (D6)**
- QA only: `UG_WW_BTMGR_RO_QA` → `Viewer`
**Fallback: direct SQL** (idempotent; refuses to change a conflicting row). It bypasses the audit and the cache
invalidation, so record it in the deployment record and **wait 30 s (or restart Central) before testing**.
```sql
SET XACT_ABORT ON;
BEGIN TRAN;
DECLARE @t TABLE (LdapGroupName nvarchar(500) NOT NULL, Role nvarchar(100) NOT NULL);
INSERT @t VALUES (N'UG_WW_APP_ADMIN_DEV', N'Administrator'), (N'UG_WW_APP_IDE_DEV', N'Designer'), (N'UG_WW_APP_IDE_RO_DEV', N'Viewer');
-- Only if D6's group exists: INSERT @t VALUES (N'UG_WW_APP_DEPLOY_DEV', N'Deployer');
-- QA: _QA suffix, plus (N'UG_WW_BTMGR_RO_QA', N'Viewer').
IF EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m JOIN @t t ON t.LdapGroupName = m.LdapGroupName WHERE m.Role <> t.Role)
THROW 50001, 'LdapGroupMappings maps one of these groups to a different role; resolve by hand.', 1;
INSERT dbo.LdapGroupMappings (LdapGroupName, Role)
SELECT t.LdapGroupName, t.Role FROM @t t
WHERE NOT EXISTS (SELECT 1 FROM dbo.LdapGroupMappings m WHERE m.LdapGroupName = t.LdapGroupName);
COMMIT;
SELECT Id, LdapGroupName, Role FROM dbo.LdapGroupMappings ORDER BY Id;
```
**Names [V]:**
- Table `LdapGroupMappings`.
- Columns `LdapGroupName` `nvarchar(500)` and `Role` `nvarchar(100)` (`SecurityConfiguration.cs:16-24`).
- Role values are the `Security/Roles.cs:45-56` constants.
The `SCADA-*` seed rows (Ids 1–5) name GLAuth groups that don't exist in NAM, so they grant nothing; leave them.
---
## 5. Decisions (owner or IT): settle before the cutover (§7)
| # | Decision | Options | Recommendation |
|---|---|---|---|
| D1 | **Service account** (owner request: one dedicated LDAP-lookup account shared by the four services, **scadaproj#58**) | **Target:** the dedicated account from scadaproj#58, shared across all four apps; one account for DEV and QA unless IT prefers one per tier. **Interim (DEV only):** `svc_igntadmin@nam.zimmer.com`, **only with the Ignition team's agreement**. It is their admin account, and their rotations or lockouts would hit us. | **Target from scadaproj#58, a QA prerequisite.** **Add to the scadaproj#58 ask: a no-lockout policy (or a very high threshold) on the account**, because it is shared by four apps and two tiers and a wrong secret costs 2 bad binds per login (§1.3). `BindMode=ServiceAccount` everywhere. Rotation = re-seed `nam-ldap-svc-password` in every store **and restart each app**, since values resolve only at startup (§7.1). |
| D2 | **Turn login on where DEV is deliberately open** (HG; mxgw = mxaccessgw#231 item 3; ScadaBridge) | Turn it on / leave DEV open | **Turn it on.** DEV is where problems should surface. |
| D3 | **Automation identity** (CLI, deploy and drill scripts). **QA:** today GLAuth `multi-role`. **DEV:** today the login-off dev user (`ManagementAuthenticator.cs:63-71`), which stops working the moment DEV ScadaBridge login turns on. The same applies to OtOpcUa scripts if DEV `Security__Auth__DisableLogin=true` (Phase 0). | (a) A NAM *user-type* identity per tier in `APP_ADMIN_<T>` + `APP_IDE_<T>` + `APP_DEPLOY_<T>`, password in the Mac Keychain. The scadaproj#58 account is lookup-only and should not double as a login unless IT agrees. (b) An interim DEV identity (a named admin's account) for DEV only. | **(a). A precondition of the DEV ScadaBridge step** (or (b) as a DEV interim) **and of the QA ScadaBridge step.** Also of the OtOpcUa step if Phase 0 finds DEV OtOpcUa login off. |
| D4 | **TLS trust** | (a) Install `zimmer-rootca` in `LocalMachine\Root`, `AllowInsecure=false`. (b) `AllowInsecure=true`: **accepts any certificate** (`ZB.MOM.WW.Auth.Ldap/Internal/NovellLdapConnection.cs:41-46`), so an active man-in-the-middle can read the service-account **and every user's** password. (c) Pin: in OtOpcUa, HG and ScadaBridge this is code-only (`ServerCertificateValidationCallback`), but **mxgw has a config-bound strict pin**, `MxGateway:Ldap:ServerCertificateThumbprintSha256` (`LdapServerCertificatePin.cs:9-21`). | **(a).** (b) is DEV only and short-lived, if at all. For mxgw: **remove any GLAuth pin** left in config (§7.3), and optionally pin the NAM DC leaf thumbprints. The certs expire 2027-03, so a pin must be renewed. |
| D5 | **Plant-floor and site groups** | now / later | **Later** (`ldap_mapping.md` §6). |
| D6 | **New group `UG_WW_APP_DEPLOY_{DEV,QA}`** (ScadaBridge `Deployer`) | ticket it / the rejected multi-role code (§3.1) | **Ticket it. Blocks the ScadaBridge step on each tier.** |
| D7 | **Group membership and exact names** | — | **Per person:** `dotnet run --project tools/LdapProbe -- --insecure --svc '<svc account UPN>' -u <sAMAccountName>` (service-account path; prompts for the service password). Or, as the person, `--insecure --self-bind -u <sAMAccountName>` (binds as them; `--svc` is ignored). **Families:** `-g 'UG_WW_*'`. Missing people get a ServiceNow membership request, not a wider mapping. Copy group names exactly (§2B). |
---
## 6. The LDAP block (each app's section)
The sections are: HG `Ldap`, mxgw `MxGateway:Ldap`, ScadaBridge `ScadaBridge:Security:Ldap`
(`Security/ServiceCollectionExtensions.cs:24`), OtOpcUa `Security:Ldap`.
```jsonc
{
"Enabled": true,
"Server": "uswap-vdc-nam1.nam.zimmer.com",
"Port": 636,
"Transport": "Ldaps",
"AllowInsecure": false, // D4
"FallbackServers": [ "uswap-vdc-nam2.nam.zimmer.com:636" ], // OtOpcUa: only with the §2 build
"SearchBase": "DC=nam,DC=zimmer,DC=com",
"ServiceAccountDn": "<svc account UPN from scadaproj#58>", // D1; DEV interim: svc_igntadmin@nam.zimmer.com (Ignition agreement)
"ServiceAccountPassword": "${secret:nam-ldap-svc-password}",
"UserNameAttribute": "sAMAccountName",
"DisplayNameAttribute": "name",
"GroupAttribute": "memberOf",
"BindMode": "ServiceAccount"
}
```
mxgw only: no `ServerCertificateThumbprintSha256` unless D4 chooses to pin the NAM DCs.
The env form follows the usual pattern, e.g. `Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636`.
---
## 7. Runbook
**Order:**
1. **DEV:** HistorianGateway (**the canary**, §7.6 rule 6) → MxAccessGateway → ScadaBridge (after §2A, D3, D6)
→ OtOpcUa (after §2).
2. A soak of a few working days with real logins.
3. **QA in the same order.** QA also needs **scadaproj#58** (QA never runs on `svc_igntadmin`). If DEV used the
interim account, switch DEV to the scadaproj#58 account during the soak, so that QA repeats a configuration
already proven on DEV.
**One app at a time:** edit, restart, verify, then the next.
**Remote execution only within the `winrm` deploy scope:** preflight, backup, edit, restart, verify, rollback.
Files go over SMB and SQL runs from the Mac. servecli is retired.
**Restart rules (review M4).** This is a **config-only** change: the same binary, no migration, no wire change.
So restarts are **rolling, standby first**:
| Pair | How | Why |
|---|---|---|
| **OtOpcUa QA pair** | Restart the standby (ServiceLevel 240/90), confirm it is healthy and back at its level, then the primary. The primary's restart is a failover; ScadaBridge's OPC UA client follows ServiceLevel, as WarmCanary showed. | The stop-both/start-both rule in OtOpcUa's records is for binary/schema upgrades. Config-only allows rolling [I: confirm against OtOpcUa's deploy notes in Phase 0]. **A bad LDAP key stops a node booting**, which is why the standby goes first. |
| **ScadaBridge QA centrals** | Standby central first, confirm it rejoins (one active, `/health/active`), then the active one, which gives a takeover of about 8–11 s. **Sites are not restarted.** | Central LDAP config only. The LocalDb stop-together rule applies to the *site* pairs' sync, which this change doesn't touch. The stop-together rule for binaries (§2A) does not apply. |
| **Single nodes** (DEV; HG/mxgw on each QA box) | One restart each. | — |
A pair restarted together **is** an outage of its data plane (OPC UA or the site data path). Don't do it for this
change.
### 7.0 Phase 0 — preflight (read-only; deploy scope; per box vd03, vq03, vq04)
1. **DC reachability:** `Test-NetConnection` to nam1 and nam2 on 636 and 389.
2. **TLS:** check the chain to nam1:636 (`SslStream`, or `certutil -verify`) and whether `zimmer-rootca` is in
`LocalMachine\Root`. This settles D4.
3. **Service-account bind, once** from the Mac:
`LDAPTLS_REQCERT=never ldapsearch -H ldaps://uswap-vdc-nam1.nam.zimmer.com:636 -x -D '<svc account UPN>' -W -b "DC=nam,DC=zimmer,DC=com" -s base dn`
4. **Current config per app, kept for rollback:**
- hash the appsettings;
- export the NSSM/service env;
- list every `Ldap__*`, `*__Ldap__*`, `*DisableLogin*`, `*__GroupToRole__*` entry;
- record **OtOpcUa's `Security__Auth__DisableLogin`** on all three boxes;
- record **mxgw's environment name** (it decides the overlay file; vd03 can't be Production while
`DisableLogin=true`);
- record **mxgw's `ServerCertificateThumbprintSha256`** (a leftover GLAuth pin blocks NAM);
- record **mxgw's base `GwAdmin`/`GwReader` entries**.
5. **QA mxgw `dc=zb` vs `dc=scadabridge`:** does a test login work today? This decides the rollback target.
6. **Secrets per app:**
- the store path, master-key source and KEK location for the §7.1 fallback. The family default is
`MasterKey:Source=Environment` / `ZB_SECRETS_MASTER_KEY`; file-based KEKs exist on some boxes. Record which
each app uses; don't assume.
- **ScadaBridge Central `Secrets:Store:Provider`.** QA is the shared SQL store (2026-11-13 QA pack §1); DEV is
[I] `Sqlite`.
- **OtOpcUa `Secrets:Replication:Enabled`** (`Host/Configuration/SecretsRegistration.cs:21`). If it's on, the
QA pair shares the secret through Akka replication, and "one store per box" becomes "seed once".
- HG KEK health on vd03 (historiangw#134).
7. **OtOpcUa DB overlay:** the `LdapGroupRoleMapping` rows (§4.3).
8. **Builds:**
- the deployed OtOpcUa build contains §2 (required before **§7.5**);
- the deployed ScadaBridge build contains §2A (before §7.4);
- on QA, the GLAuth `multi-role` user still maps to a role after §2/§2A (§2.4).
### 7.1 Seed and verify the secret (before any config change)
- **Name:** `nam-ldap-svc-password`, the same in every store.
- Values resolve **only at startup**, so a rotation is re-seed every store plus restart every app (§7 restart
rules).
- An **unresolved** `${secret:}` throws `SecretNotFoundException` at boot (`SecretReferenceExpander.cs:74-77`),
so the app won't start.
- **Preferred:** each app's `/admin/secrets` page. It writes through the app's own provider, which is correct for
ScadaBridge's shared SQL store and for a replicated OtOpcUa store.
- DEV HG, mxgw and ScadaBridge are still login-off at this point. On QA, use the current GLAuth admin.
- **Fallback:** the `secret` CLI, with the store and master-key settings **from Phase 0 step 6**, per app:
```powershell
$env:Secrets__SqlitePath = "<that app's store, Phase 0>"
$env:Secrets__MasterKey__Source = "<Environment|File, Phase 0>" # family default: Environment (ZB_SECRETS_MASTER_KEY)
$env:Secrets__MasterKey__FilePath = "<kek, only if File>"
secret.exe set nam-ldap-svc-password # prompts; never pass the value as an argument
```
- **Stores:**
- HG and mxgw: per box.
- OtOpcUa: per box, or once if replication is on.
- ScadaBridge: **once per tier through Central**. The site does no LDAP.
- **Verify the VALUE before the app's restart (review M2):** read it back (`get`) and bind with it once using
LdapProbe (`--svc '<svc account UPN>'`, pasting the read-back value). One good bind proves the stored secret.
A wrong secret found here costs 1–2 bad binds, instead of 2 per user login after the restart.
### 7.2 HistorianGateway (DEV: file; QA: NSSM env). The canary.
1. Back up the file or env.
2. Write §6 and §4.1. On DEV, set `DisableLogin` to `false`. On QA, replace the `Ldap__*` entries and remove the
`SCADA-*` `GroupToRole` entries.
3. Restart `ZbHistorianGateway`. Run §7.6 **fully, including the failover check**, before touching any other app.
### 7.3 MxAccessGateway (the file for its environment name)
1. Back up the base and overlay files.
2. Write `MxGateway:Ldap` (§6) and `GroupToRole` (§4.2), and delete the base `GwAdmin`/`GwReader`.
3. **Remove the `ServerCertificateThumbprintSha256` pin**, or replace it with the NAM DC leaf thumbprints (D4).
4. On DEV, set `DisableLogin` to `false` (D2, mxaccessgw#231 item 3). If that makes the box eligible to run as
Production, record the environment name. Don't change it in this step.
5. Leave the API-key store under `data\` and the pepper env alone.
6. Restart, then run §7.6.
### 7.4 ScadaBridge (Central only)
**Preconditions:**
- §2A deployed;
- D6: `UG_WW_APP_DEPLOY_<T>` exists, has members, and is mapped;
- D3: the automation identity works. On DEV it is needed **before** login turns on, because the dev-user path
ends then.
1. Back up `appsettings.Central.json` and the env.
2. Add the role mappings (§4.4, CLI preferred).
3. Write `ScadaBridge:Security:Ldap` (§6).
4. On DEV, **remove BOTH** `ScadaBridge:Security:Auth:DisableLogin` and `AllowOutsideDevelopment`. Removing only
the second while `DisableLogin=true` makes `DisableLoginGuard` crash-loop the service.
5. On QA, replace the GLAuth values.
6. Restart: DEV, the single central. QA, rolling, standby central first (§7 restart rules).
7. Re-point every script that used `multi-role` (or relied on login-off) at the D3 identity: `apply.sh`, the
drill and deploy packs. Confirm `/auth/token` works with it.
### 7.5 OtOpcUa AdminUI (service env; only after §2 is deployed)
1. Back up the env.
2. Add §6 as `Security__Ldap__*` and the §4.3 `GroupToRole`. Keep the files' `Transport`/`DevStubMode`.
3. If `Security__Auth__DisableLogin=true`, set it to `false`, and if scripts relied on it, have D3 in place first.
4. Restart: DEV, the single node. QA, **rolling, standby first** (§7 restart rules). Watch ServiceLevel and the
ScadaBridge `WarmCanary` connection across the primary's restart.
5. **Existing sessions:** AdminUI cookies and bearer tokens issued under GLAuth stay valid after the switch until
they expire. Either:
- invalidate them (restart both nodes, if the cookie and data-protection keys are per process [I], or rotate
the signing or data-protection key); or
- record their remaining lifetime and accept it.
ScadaBridge needs no such step: its group refresh ends sessions whose roles change
(`DirectoryGroupRefresher.cs:141-142`).
### 7.6 Verify (each app, right after its restart)
1. **Negative control first:** a bogus user and password. Expect a clean "invalid / not authorized" and **no**
`failed system-side` warning.
- A `service-bind`-stage warning means the stored service secret is wrong. **Stop: every further attempt costs
the shared account 2 bad binds.**
- A `connect`-stage warning means network or TLS.
- (An unresolved `${secret:}` never gets this far; the app fails at boot with `SecretNotFoundException`.)
- (OtOpcUa logs these lines only with the §2 logger fix.)
2. **One real login per mapped role:** Administrator, Designer, Viewer, and Deployer for ScadaBridge. The UI shows
exactly that role, and the user typed a **bare sAMAccountName** (§1.3).
3. **A user whose groups map to no role is refused**, on the login page **and** at the token endpoint where one
exists (OtOpcUa and ScadaBridge `/auth/token`). This is the §3.2 standard: in OtOpcUa and ScadaBridge it holds
only with the §2/§2A builds; mxgw and HG already refuse.
4. **`${secret:}` proof:** no `*ServiceAccountPassword` sits in env, and the stored value was bind-verified
(§7.1).
5. **Failover:**
- On HG (the canary), once per tier: block nam1:636 briefly. Login still works and the log shows `sticky
preference moved`; then restore.
- **OtOpcUa** gets its own nam1→nam2 check (acceptance A2).
6. **Stop rule (review M2):** stop the cutover on **2 failed logins attributable to the service account, across
all apps, within 15 minutes.** Count `service-bind` warnings, not user typos. Diagnose before any retry; ask IT
to check the account's lockout state.
### 7.7 Rollback (per app)
- Restore the backed-up files or env and restart. On QA, use the same rolling order: standby first.
- mxgw: also restore its base `appsettings.json` (the `GwAdmin`/`GwReader` entries) and any pin.
- **Point the scripts back at `multi-role`** (QA), or back at login-off (DEV ScadaBridge).
- If the shared account got locked, **IT clears the lockout.** Until then, every NAM-backed app is down for login;
roll those apps back to GLAuth or login-off first.
- The seeded secrets are inert once nothing references them. The ScadaBridge `UG_WW_*` mapping rows grant nothing
while login is off or on GLAuth.
- **Keep** the §2/§2A/§2B builds. The zero-role refusal is the family standard and stands on its own.
---
## 8. Acceptance
- **A1:** the §2 offline tests pass:
- failover on the redundant GLAuth pair with the library's log line;
- zero-role refusal (cookie + token, including the mapper-throws case);
- a DEV-shaped config boots;
- the reflection guard fails under its mutation.
- The §2A tests pass too.
- **A2:** on each tier, the OtOpcUa AdminUI login survives nam1 being blocked, through nam2.
- **A3:** every §7.6 check passes for each tier and app.
- **A4:** no app maps any group to two roles, and no stale GLAuth mapping remains. Check the **effective**
config: env plus all overlays plus base, including mxgw's `GwAdmin`/`GwReader`, and the ScadaBridge table.
- **A5:** automation runs on the D3 identity. Nothing uses `multi-role`, or relies on login-off, on a cut-over
app.
- **A6:** QA (and DEV after the soak) binds as the scadaproj#58 account and never as `svc_igntadmin`.
## 9. Open questions
1. ~~Does OtOpcUa run the library's validator?~~ **Answered: no** (§2.1 item 4). The validation is ported
selectively (§2.2 item 4).
2. Does QA mxgw login work today (`dc=zb` vs `dc=scadabridge`)? Phase 0 step 5.
3. What is the HG KEK state on vd03? (historiangw#134)
4. Is OtOpcUa's AdminUI cookie or data-protection key per process or shared? That decides the §7.5 step 5 method
[I].
5. Does OtOpcUa's deploy procedure allow a rolling restart for config-only changes? Confirm against its records
(§7 restart rules).
6. Do the `UG_WW_*` memberships and exact names still match 2026-08-12? D7 re-enumerates.
7. ignitionoee: still on hold.
## 10. Corrections from the 2026-08-13 plan
| 08-13 plan said | Fact (2026-11-15) |
|---|---|
| Phase 1: redeploy to Auth 0.2.1 | Every build already runs 0.4.1 (ScadaBridge 0.5.0). Phase deleted. |
| Preflight over servecli `:2222` | servecli is retired; the winrm deploy scope plus SMB plus SQL replace it. |
| mxgw on vd03 has LDAP login on | `DisableLogin=true` (and so it isn't running as Production). |
| One target block for all four apps | OtOpcUa drops `FallbackServers` and three more options, and logs nothing. Fixed in §2. |
| A zero-group user is denied in every app | **Only mxgw and HG refuse.** OtOpcUa and ScadaBridge sign in with no roles. Fixed in §2/§2A; the standard is §3.2. |
| ScadaBridge `INSERT INTO LdapGroupMapping`, two rows for `APP_IDE` | The table is `LdapGroupMappings`, unique `LdapGroupName`. One-group-one-role plus the new D6 group; CLI preferred over SQL. |
| `ldap_mapping.md`: ScadaBridge can grant two roles per group | It can't. Rewritten as the canonical matrix. |
| Seed the ScadaBridge central and site stores with the CLI | Seed once through Central `/admin/secrets`; the site does no LDAP. |
| Stop after 2 failures; every failed login costs both DCs | A wrong *user* password costs 1 bind. A wrong *service* secret costs 2 per login and locks the shared account. New stop rule, a no-lockout ask and value verification. |
| Service account `svc_igntadmin` now, dedicated "later" | Target: scadaproj#58, a QA prerequisite; `svc_igntadmin` is a DEV interim only, with Ignition's agreement. |
| OtOpcUa before ScadaBridge | OtOpcUa last, after its fix. |
| (silent) automation | DEV and QA automation lose their identity on cutover. D3 is a precondition. |
| (silent) restarts | Rolling, standby first, for config-only changes. Restarting a pair together is a data-plane outage. |
## 11. Review 2026-11-15: findings and disposition
| # | Finding | Disposition |
|---|---|---|
| B1 | OtOpcUa and ScadaBridge sign in users with zero mapped roles (OtOpcUa even when the mapper throws); `AuthenticatedRead`/bare `[Authorize]` then grant read access to any AD user | **Applied.** New standard §3.2; OtOpcUa fix §2.2 item 2 (refuses on throw too, reversing `Login_when_role_mapper_throws_signs_in_with_no_role_claims`); ScadaBridge Phase A2 §2A with issue text and a deploy; Viewer semantics §3.2; §7.6 step 3 and `ldap_mapping.md` §1 rewritten. |
| B2 | Calling the library validator would stop OtOpcUa booting on DEV/QA (`ServiceAccountDn` required) | **Applied.** §2.2 item 4 ports only the `FallbackServers` and BindMode/UserBindTemplate rules, with a DEV-shaped boot test. Open question 1 answered (§2.1 item 4). |
| M1 | No logger passed to the library service | **Applied.** §2.2 item 3. |
| M2 | Lockout on the shared account; §1.3 had it backwards | **Applied.** §1.3 table; D1 no-lockout ask; §7.1 value verification; HG as canary; §7.6 rule 6 stop rule. |
| M3 | DEV automation breaks when DEV login turns on | **Applied.** D3 covers DEV (dev-user path, `ManagementAuthenticator.cs:63-71`); a precondition in §7.4/§7.5. |
| M4 | "Data plane unaffected" was misleading; pairs restarted together are an outage | **Applied.** §7 restart rules: rolling, standby first, for config-only changes. The OtOpcUa rolling allowance is [I] pending Phase 0 / open question 5. |
| M5 | HG group match is case-sensitive after binding | **Applied.** §2B (optional Phase A3) with issue text; exact names from LdapProbe in §4 and D7. |
| M6 | mxgw config-bound certificate pin | **Applied.** Phase 0 step 4; §7.3 step 3; D4 corrected. |
| m | §2.4 / §7.0 cross-references | **Applied** (→ §7.5). |
| m | Unresolved `${secret:}` fails at boot | **Applied** (§7.1, §7.6). |
| m | Rotation needs restarts | **Applied** (D1, §7.1). |
| m | `AcceptDomainQualifiedNames` only applies in User mode | **Applied** (§1.3; users type the bare sAMAccountName). |
| m | D7 command was wrong | **Applied** (`-u` with `--svc`, or `--self-bind -u`). |
| m | Secrets CLI hardcoded `File` | **Applied** (Phase 0 step 6 supplies the values; default noted). |
| m | OtOpcUa secret replication | **Applied** (Phase 0 step 6; §7.1 stores). |
| m | mxgw vd03 can't be Production with `DisableLogin` | **Applied** (§1.2; Phase 0 step 4; §7.3 step 4). |
| m | mxgw base `GwAdmin`/`GwReader` merge | **Applied** (§4.2 delete from base; A4 checks the effective config). |
| m | Existing OtOpcUa sessions survive cutover | **Applied** (§7.5 step 5; ScadaBridge self-ends via refresh). |
| m | Rollback gaps | **Applied** (§7.7: scripts, AD lockout, rolling on QA). |
| m | `AllowInsecure` accepts any certificate | **Applied** (D4(b)). |
| m | Stale OtOpcUa validator comment | **Applied** (§2.1 item 5, §2.2 item 7). |
| m | ScadaBridge SQL bypasses audit and cache | **Applied** (§4.4: CLI/UI preferred; SQL fallback with a 30 s wait). |
| n | `LdapOptions` doc default `10.100.0.35` vs code `localhost` | **Applied** (§2.1 item 5). |
| n | Reflection test: compare lists by content, both directions | **Applied** (§2.2 item 5). |
| n | §4.4 DEPLOY row conditional | **Applied.** |