docs(plan): PROD install gets the NAM AD LDAP settings (§3A): block, _PROD mappings, secret before first start, D-P5..D-P7

Claude-Session: https://claude.ai/code/session_01GT67AD5mc2BnaaAfY3hqqQ
This commit is contained in:
Joseph Doherty
2026-11-19 16:24:04 -05:00
parent 9789b01acd
commit 8e415212fa
@@ -3,8 +3,9 @@
**State: PLAN ONLY. Nothing has been executed, and no prod box has been touched.** This is the first production
tier for the family. Until now `deployment.md` said "no production box has been identified".
**The DEPLOY HOLD still applies to QA and PROD** (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every
phase below that touches QA or a prod box needs its own explicit owner go.
**The DEPLOY HOLD still applies to PROD** (owner, 2026-11-17). DEV and QA have both run the candidate builds with NAM AD
login since 2026-11-19 (QA window approved that day). Every phase below that touches a prod box needs its own explicit
owner go. **LDAP settings for PROD: §3A** (added 2026-11-19 after the DEV and QA cutovers).
## 1. Target topology
@@ -28,7 +29,8 @@ boundary for the first time. It is the main new risk in this install.
which refuse zero-role logins) and mxgw `e74918cc`. Prod gets the **same zips, byte-identical** to the ones
that soaked on QA.
- **D-P3 NAM AD over LDAPS from day one.** This uses `_PROD` groups only and separate PROD accounts. There is
no GLAuth in prod.
no GLAuth in prod. The settings, the secret handling and the open bind-account / Deployer / cookie decisions
(D-P5–D-P7) are in §3A.
- **D-P4 content: a signed config bundle from QA**, limited to the machines selected for prod. Prod
connections are re-pointed at the prod gateways, and prod mints its own inbound API keys.
@@ -73,7 +75,7 @@ for `sqlvd03`). Whether `adm_dohertj2` is a local admin on the wonder prod boxes
**DELMIA prod is `dnc-app-vp01`** (10.201.207.106). For Q8, that box is the prod Delmia caller of
ScadaBridge's `DelmiaRecipeDownload`. Prod runs an older Downloader than QA (`delmia` project memory), so check
the prod integration's shape before you assume it matches QA's.
| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. | Phases 4–5 |
| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? It decides the cookie setting (D-P7, §3A). LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. DEV and QA already had the root CA (thumbprint `78D941D7…`, exp 2036) and validated nam1/nam2 over TLS 1.3. | Phases 4–5 |
| Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the `SiteId`? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 |
| Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 |
@@ -87,6 +89,8 @@ the prod integration's shape before you assume it matches QA's.
- G3 `UG_WW_APP_DEPLOY_PROD`;
- a **membership review** of `UG_WW_APP_ADMIN_PROD`, `APP_IDE_PROD`, `APP_IDE_RO_PROD` and `BTMGR_RO_PROD`.
These groups were built for System Platform, so mapping them grants their members app roles in prod.
Current members are in §3A.2. Also confirm no NAM group is named `SCADA-*` (ScadaBridge's seed rows).
Decide D-P5 (bind account) and D-P6 (Deployer) if A1P or G3 will not exist by go-live.
2. Extend scadaproj#58 for A1P (comment). File a `cross-repo` umbrella issue in scadaproj, plus a `type:deploy-gate`
issue in ScadaBridge and in mxaccessgw, cross-linked.
3. Firewall requests, from the §4 matrix.
@@ -114,8 +118,9 @@ the prod integration's shape before you assume it matches QA's.
4. **Soak on QA for at least 3 working days.** The gates are the QA record's set: ready 200 ×4, one active per
pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative
control).
5. Optionally, do QA's NAM cutover (plan Phases A–7) before prod. **Recommended.** QA is then the first
environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first.
5. ~~Optionally, do QA's NAM cutover before prod.~~ **Done 2026-11-19**, in the same window as the QA deploy of these
builds (record `docs/deploy/2026-11-19-qa-nam-ldap/`). The QA soak therefore covers NAM AD over LDAPS on the exact
PROD candidates; it started 2026-11-19, so the 3-working-day gate is met on 2026-11-24 at the earliest.
### Phase 2: box prep (all four, no service started)
@@ -128,6 +133,9 @@ the prod integration's shape before you assume it matches QA's.
- Optional, and recommended for parity: the `ZbOtelCollector` loopback collector (`infra/otel-collector/wonder/`).
- Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN
that the config will use.
- LDAP gate (§3A), read-only, per box: nam1 and nam2 reachable on 636; `zimmer-rootca` in `LocalMachine\Root`; an
LDAPS handshake to both validates with no policy errors (the QA `preflight.ps1` steps 1–2). Stage the Secrets CLI 0.6.6
under `E:\ApiInstall\_tools\secret-0.6.6\`, plus the SQL-aware build on the centrals.
### Phase 3: ConfigDb
@@ -139,6 +147,7 @@ the prod integration's shape before you assume it matches QA's.
3. Gate: 65 rows in `__EFMigrationsHistory`, the `zbauth` schema present (inbound API keys, Auth
`ApiKeys.SqlServer`), and the Secrets SqlServer store tables present.
4. First backup: `COPY_ONLY, CHECKSUM, COMPRESSION` + `VERIFYONLY`.
5. Insert the `_PROD` role rows by SQL (§3A.3). Gate: the rows read back; ids 1–5 (`SCADA-*`) untouched.
### Phase 4: MxAccessGateway ×4
@@ -149,9 +158,9 @@ the connection to the local PAR gateways as a normal connection edit and instanc
Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the masked env before the real run.
- Environment `Production`, which **requires login on**. LDAPS to NAM (nam1/nam2 + `FallbackServers`), the A1P
bind, and GroupToRole for `_PROD` groups only: `APP_ADMIN_PROD`→Administrator, `APP_IDE_PROD`→Viewer,
`APP_IDE_RO_PROD`→Viewer (`ldap_mapping.md` §4.1).
- Environment `Production`, which **requires login on**. The LDAP env block and `GroupToRole` from §3A.3 go into the
`-EnvFile`; delete `GwAdmin`/`GwReader` from the base `appsettings.json` before the first start.
- Install **without `-Start`**, seed `nam-ldap-svc-password` (§3A.4), review the masked env, then start.
- **Fresh prod secrets on each box:** `ZB_SECRETS_MASTER_KEY`, `MxGateway__ApiKeyPepper`. Nothing is copied from
QA. Choose one of two patterns and record the choice:
(a) **Recommended:** the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so
@@ -160,7 +169,7 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the
- Galaxy SQL and the alarm subscription per Q2. `SqlitePath` values are absolute and outside the content root.
- Mint client keys: `scadabridge-site-prod` on the PAR gateways, plus whatever Q4 decides for the app boxes.
Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else.
- Gates: dashboard login with a NAM `_PROD` user succeeds; a user with zero roles is refused; with the client
- Gates: §3A.5 (dashboard login as `dohertj2` → Administrator; bogus refused; no service-bind line); with the client
token, `galaxy-test-connection` returns `ok:true`, and a session open/close works on each box; a bogus key
gets a typed auth error; a worker session reports ready.
@@ -172,9 +181,10 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the
- the ConfigDb connection string; `Secrets:Store:Provider=SqlServer` (one store shared by both centrals);
- a **shared `JwtSigningKey`** (≥ 32 random characters), shared ASP.NET Data Protection keys, and an
`ApiKeyPepper`;
- NAM LDAPS with `AllowInsecure=false` and the A1P bind;
- GroupToRole for `_PROD` only, including `APP_DEPLOY_PROD`→Deployer (`Administrator` does not include
Deployer);
- NAM LDAPS: the §3A.3 block in each node's `appsettings.Central.json`, no `Auth` block, no LDAP env entry,
`RequireHttpsCookie` per D-P7; the secret seeded once into the shared store before the first start (§3A.4);
- roles come from the `LdapGroupMappings` rows inserted in Phase 3 (`_PROD` only, `APP_DEPLOY_PROD`→Deployer or
the D-P6 fallback; `Administrator` does not include Deployer);
- downing `leader-auto-down` (the shipped default); OTLP to loopback if Phase 2 installed the collector.
Start from the QA Central env **key names** (names only, never values) plus QA's appsettings shape. Write a
per-node diff table before anything starts.
@@ -184,8 +194,10 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the
boundary the runbook describes).
- Start order: vp03, wait for ready 200, then vp04.
- Gates: ready 200 on both; active 200 on one node and 503 on the other; `required-singletons` healthy;
`bundle-signing-policy` healthy; a NAM login works in the UI and in the CLI (`--url`, A3P or a named
deployer); a zero-role user is refused at `/auth/login`, `/auth/token` and `/management`.
`bundle-signing-policy` healthy; §3A.5 on the active central: `dohertj2` in the UI, at `/auth/token` (role claims)
and in the CLI (`--url`, `--password-file`), bogus refused, no service-bind line; then the same login against the
other central after a failover drill in Phase 8. A zero-role user is refused at `/auth/login`, `/auth/token` and
`/management` where §3A.5 item 4 has an account for it.
### Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04)
@@ -227,6 +239,128 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the
- Overview: decide whether prod gets its own Overview, or a `Prod` registry on an existing box. It is not in
scope unless the owner asks.
## 3A. LDAP settings (NAM AD) for PROD
Login is on from the first start (D-P3). This section is the PROD form of the family cutover
([`2026-11-15-nam-ldap-cutover.md`](2026-11-15-nam-ldap-cutover.md)), which already ran on DEV and QA with these
exact builds (records [`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md) and
[`docs/deploy/2026-11-19-qa-nam-ldap/`](../deploy/2026-11-19-qa-nam-ldap/README.md)). Differences from QA: PROD is
greenfield, so there is no GLAuth config to replace, and the settings are written at install time, before the first start.
### 3A.1 Decisions
| # | Decision | Recommendation |
|---|---|---|
| D-P5 | **Bind account.** PROD binds as **A1P `svc_zbmomww_prod`** (T6; no lockout, read-only). DEV and QA run on the interim `svc_igntadmin`, which belongs to the Ignition team. | **Wait for A1P.** Using `svc_igntadmin` in PROD means one Ignition-owned password (and one lockout) spans every tier and the Ignition PROD stack. Only with an explicit owner go **and** the Ignition team's OK; then the A1P swap is a re-seed of `nam-ldap-svc-password` in every PROD store plus a rolling restart. |
| D-P6 | **ScadaBridge Deployer.** `UG_WW_APP_DEPLOY_PROD` (G3, T6). | **Wait for G3.** Fallback, as on DEV/QA: `UG_WW_HIST_ADMIN_PROD`→Deployer. Note it is **not** the same membership as `APP_ADMIN_PROD` in PROD (4 vs 5 members on 2026-11-19: `millerh5` is admin but not hist-admin), so with the fallback that admin cannot deploy. |
| D-P7 | **Cookies over plain HTTP.** Each app's auth cookie must be `Secure` only when the endpoint is HTTPS. | If Q7 delivers certs: HTTPS endpoints and `RequireHttpsCookie=true`. Without TLS: `RequireHttpsCookie=false` on mxgw and ScadaBridge (as on QA), recorded as a known gap. |
### 3A.2 Directory facts (2026-11-19 dump; re-verify in Phase 0)
| Group | Members | ScadaBridge | MxAccessGateway |
|---|---|---|---|
| `UG_WW_APP_ADMIN_PROD` | 5 (chamalas, millerh5, marting4, dohertj2, Luis Bastidas) | Administrator | Administrator |
| `UG_WW_APP_IDE_PROD` | 4 (chamalas, marting4, dohertj2, Luis Bastidas) | Designer | Viewer |
| `UG_WW_APP_DEPLOY_PROD` | **does not exist yet** (G3) | Deployer | — |
| `UG_WW_APP_IDE_RO_PROD` | 1 (oswaltk) | Viewer | Viewer |
| `UG_WW_BTMGR_RO_PROD` | 4 (benjamd2, marting4, oswaltk, Luis Bastidas) | Viewer | — |
| `UG_WW_HIST_ADMIN_PROD` | 4 | (D-P6 fallback only) | — |
`UG_WW_BTMGR_{BE,MC}_PROD` and `UG_WW_CITRIX_PROD` are **never** mapped here (§4.2 rows wait for later). Each PROD
deployment lists **only `_PROD` groups**; a `_DEV`/`_QA` name in a PROD config is a defect. The T6 membership review
must finish before the mapping goes live: `BTMGR_RO_PROD` gives 4 people ScadaBridge Viewer.
**Test identity:** `NAM\dohertj2` is in `APP_ADMIN_PROD` and `APP_IDE_PROD`, so it is the PROD login probe (expected
roles: ScadaBridge Administrator+Designer, plus Deployer once G3 or D-P6 lands; mxgw Administrator). Its password is in
the Keychain as `winrm-cli/vq03`. `adm_dohertj2` is the box-admin account for winrm/SMB only, never a login probe.
### 3A.3 The block (identical on every app; only the key prefix differs)
| Setting | Value |
|---|---|
| `Enabled` | `true` |
| `Server` / `Port` / `Transport` | `uswap-vdc-nam1.nam.zimmer.com` / `636` / `Ldaps` |
| `FallbackServers[0]` | `uswap-vdc-nam2.nam.zimmer.com:636` |
| `AllowInsecure` | `false` (needs `zimmer-rootca` in `LocalMachine\Root`; Phase 2 gate) |
| `SearchBase` | `DC=nam,DC=zimmer,DC=com` |
| `ServiceAccountDn` | `svc_zbmomww_prod@nam.zimmer.com` (A1P; D-P5) |
| `ServiceAccountPassword` | `${secret:nam-ldap-svc-password}`, never a literal |
| `UserNameAttribute` / `DisplayNameAttribute` / `GroupAttribute` | `sAMAccountName` / `name` / `memberOf` |
| `BindMode` | `ServiceAccount` |
**MxAccessGateway (×4): NSSM env**, through `Install-MxGatewayService.ps1 -EnvFile` (the QA pattern; the upgrade script
never writes the env, so it survives every redeploy):
```
MxGateway__Ldap__Enabled=true
MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com
MxGateway__Ldap__Port=636
MxGateway__Ldap__Transport=Ldaps
MxGateway__Ldap__AllowInsecure=false
MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636
MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com
MxGateway__Ldap__ServiceAccountDn=svc_zbmomww_prod@nam.zimmer.com
MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password}
MxGateway__Ldap__UserNameAttribute=sAMAccountName
MxGateway__Ldap__DisplayNameAttribute=name
MxGateway__Ldap__GroupAttribute=memberOf
MxGateway__Ldap__BindMode=ServiceAccount
MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_PROD=Administrator
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_PROD=Viewer
MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_PROD=Viewer
MxGateway__Dashboard__RequireHttpsCookie=<per D-P7>
```
Plus, in the base `app\appsettings.json`, **delete `MxGateway:Dashboard:GroupToRole` (`GwAdmin`/`GwReader`)** before the
first start: configuration merges the file map with the env map, so the GLAuth names would otherwise stay mapped. The
upgrade script carries the edited file forward on later upgrades.
**ScadaBridge Central (×2): `app\appsettings.Central.json`**, section `ScadaBridge:Security:Ldap` (the QA shape; each
node's file is written separately, never copied over the other), with no `Auth` block (`DisableLogin` /
`AllowOutsideDevelopment` absent) and `ScadaBridge:Security:RequireHttpsCookie` per D-P7. **No
`ScadaBridge__Security__Ldap__*` env entry**: an env password would override the `${secret:}` reference. Sites have no
login and get no LDAP settings.
**ScadaBridge role rows**, in `dbo.LdapGroupMappings` (the central database, not config). With login on from the first
start nobody can sign in until they exist, so insert them **by SQL after Phase 3** with the idempotent script
`docs/deploy/2026-11-19-vd03-nam-ldap/sb-role-mappings.ps1` retargeted to these rows:
`UG_WW_APP_ADMIN_PROD`→Administrator, `UG_WW_APP_IDE_PROD`→Designer, `UG_WW_APP_IDE_RO_PROD`→Viewer,
`UG_WW_BTMGR_RO_PROD`→Viewer, `UG_WW_APP_DEPLOY_PROD`→Deployer (or the D-P6 fallback). Later changes go through the CLI
(audited). The migration seeds five `SCADA-*` rows (ids 1–5); they grant nothing unless a NAM group with that exact
name exists, which Phase 0 checks (`LdapProbe -g 'SCADA-*'` returns none).
### 3A.4 The secret, before the first start
Each app's pre-host expander fails closed on a missing `${secret:}`, so `nam-ldap-svc-password` must be in the store
before the service is started for the first time:
| Store | Where | How |
|---|---|---|
| mxgw ×4 | each box's `Secrets__SqlitePath` (absolute, outside the content root), KEK per that service's env | `Install-MxGatewayService.ps1` **without `-Start`**, then `seed-secret.ps1` (the QA one; it reads the store path and KEK from the service env), then start |
| ScadaBridge centrals | the shared SQL Server store (`zbsecrets`, `Secrets:Store:Provider=SqlServer`), **once** | the SQL-aware Secrets CLI build (the headless 0.6.6 CLI writes SQLite only; QA finding 2). Both centrals must hold the **same KEK file** (check the hash on both) |
The value goes Keychain → 0600 temp file → SMB → ACL'd file → CLI stdin → deleted; read back and compared on the box.
It is never on argv, in chat or in a log. Use the **Secrets CLI 0.6.6** (stdin); 0.6.2 stores "-" literally.
### 3A.5 Verification and stop rule (after each app's first start)
1. Anonymous `GET /` → 302 `/login`.
2. A bogus user is refused, with no `system-side` / service-bind line in the log.
3. `dohertj2` signs in with the expected roles (ScadaBridge: cookie, `/auth/token` claims, and the CLI over HTTP Basic
against the active central; mxgw: dashboard).
4. Zero-role refusal: needs a NAM account the operator controls that is in no `_PROD` group. If there is none, rely on
the DEV/QA proofs of #144/#145 and say so in the record.
5. Logs: the app's own log or the `ZbOtelCollector` export if Phase 2 installed it (HG-style empty app logs happened on DEV).
**Stop at the first service-account bind failure** (the A1P account is no-lockout, but stop anyway and check with IT),
not the second. No LDAP DC-failover drill (blocking nam1 or pointing at a closed port) and no ad-hoc firewall rules for
testing (owner, 2026-11-19, for the DEV/QA cutover; carried over here). The §4 inbound rules are a separate, planned change.
### 3A.6 Rollback
PROD is greenfield until Phase 7 step 5, so an LDAP problem before then means: stop the service, fix the env, file or
rows, start again. There is no GLAuth to fall back to and none is installed.
## 4. Port and firewall matrix (to request in Phase 0)
| From | To | Port | Purpose |
@@ -264,6 +398,8 @@ the existing site gRPC listener.
| The central↔site boundary (`8083`) is blocked or filtered in one direction only. QA never exercised it, because it was co-located | Phase 0 firewall ticket; Phase 6 round-trip gate in **both** directions |
| One machine driven by two stacks (Q9) | Undeploy from QA before the prod deploy |
| `_PROD` groups grant wider access than intended | T6 membership review before mapping |
| A service-account lockout breaks every app | A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule) |
| A service-account lockout breaks every app | A1P is no-lockout; stop at the **first** service-account bind failure (§3A.5). Using the Ignition-owned `svc_igntadmin` in PROD (D-P5 fallback) would tie PROD to Ignition's password |
| `${secret:nam-ldap-svc-password}` missing at first start (the expander fails closed and the service will not boot) | Seed before the first start (§3A.4); install mxgw without `-Start` |
| A `_DEV`/`_QA` group name in a PROD config grants lower-tier admins PROD roles | §3A.2 rule; review the env file and the role rows against §3A.3 before the start |
| The mxgw upgrade script aborts on a slow start | Phase 1 step 1 fix |
| The security software flags remote exec | Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert |