diff --git a/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md index 1a56889..6feff6e 100644 --- a/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md +++ b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md @@ -3,8 +3,9 @@ **State: PLAN ONLY. Nothing has been executed, and no prod box has been touched.** This is the first production tier for the family. Until now `deployment.md` said "no production box has been identified". -**The DEPLOY HOLD still applies to QA and PROD** (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every -phase below that touches QA or a prod box needs its own explicit owner go. +**The DEPLOY HOLD still applies to PROD** (owner, 2026-11-17). DEV and QA have both run the candidate builds with NAM AD +login since 2026-11-19 (QA window approved that day). Every phase below that touches a prod box needs its own explicit +owner go. **LDAP settings for PROD: §3A** (added 2026-11-19 after the DEV and QA cutovers). ## 1. Target topology @@ -28,7 +29,8 @@ boundary for the first time. It is the main new risk in this install. which refuse zero-role logins) and mxgw `e74918cc`. Prod gets the **same zips, byte-identical** to the ones that soaked on QA. - **D-P3 NAM AD over LDAPS from day one.** This uses `_PROD` groups only and separate PROD accounts. There is - no GLAuth in prod. + no GLAuth in prod. The settings, the secret handling and the open bind-account / Deployer / cookie decisions + (D-P5–D-P7) are in §3A. - **D-P4 content: a signed config bundle from QA**, limited to the machines selected for prod. Prod connections are re-pointed at the prod gateways, and prod mints its own inbound API keys. @@ -73,7 +75,7 @@ for `sqlvd03`). Whether `adm_dohertj2` is a local admin on the wonder prod boxes **DELMIA prod is `dnc-app-vp01`** (10.201.207.106). For Q8, that box is the prod Delmia caller of ScadaBridge's `DelmiaRecipeDownload`. Prod runs an older Downloader than QA (`delmia` project memory), so check the prod integration's shape before you assume it matches QA's. -| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. | Phases 4–5 | +| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? It decides the cookie setting (D-P7, §3A). LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. DEV and QA already had the root CA (thumbprint `78D941D7…`, exp 2036) and validated nam1/nam2 over TLS 1.3. | Phases 4–5 | | Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the `SiteId`? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 | | Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 | @@ -87,6 +89,8 @@ the prod integration's shape before you assume it matches QA's. - G3 `UG_WW_APP_DEPLOY_PROD`; - a **membership review** of `UG_WW_APP_ADMIN_PROD`, `APP_IDE_PROD`, `APP_IDE_RO_PROD` and `BTMGR_RO_PROD`. These groups were built for System Platform, so mapping them grants their members app roles in prod. + Current members are in §3A.2. Also confirm no NAM group is named `SCADA-*` (ScadaBridge's seed rows). + Decide D-P5 (bind account) and D-P6 (Deployer) if A1P or G3 will not exist by go-live. 2. Extend scadaproj#58 for A1P (comment). File a `cross-repo` umbrella issue in scadaproj, plus a `type:deploy-gate` issue in ScadaBridge and in mxaccessgw, cross-linked. 3. Firewall requests, from the §4 matrix. @@ -114,8 +118,9 @@ the prod integration's shape before you assume it matches QA's. 4. **Soak on QA for at least 3 working days.** The gates are the QA record's set: ready 200 ×4, one active per pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative control). -5. Optionally, do QA's NAM cutover (plan Phases A–7) before prod. **Recommended.** QA is then the first - environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first. +5. ~~Optionally, do QA's NAM cutover before prod.~~ **Done 2026-11-19**, in the same window as the QA deploy of these + builds (record `docs/deploy/2026-11-19-qa-nam-ldap/`). The QA soak therefore covers NAM AD over LDAPS on the exact + PROD candidates; it started 2026-11-19, so the 3-working-day gate is met on 2026-11-24 at the earliest. ### Phase 2: box prep (all four, no service started) @@ -128,6 +133,9 @@ the prod integration's shape before you assume it matches QA's. - Optional, and recommended for parity: the `ZbOtelCollector` loopback collector (`infra/otel-collector/wonder/`). - Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN that the config will use. +- LDAP gate (§3A), read-only, per box: nam1 and nam2 reachable on 636; `zimmer-rootca` in `LocalMachine\Root`; an + LDAPS handshake to both validates with no policy errors (the QA `preflight.ps1` steps 1–2). Stage the Secrets CLI 0.6.6 + under `E:\ApiInstall\_tools\secret-0.6.6\`, plus the SQL-aware build on the centrals. ### Phase 3: ConfigDb @@ -139,6 +147,7 @@ the prod integration's shape before you assume it matches QA's. 3. Gate: 65 rows in `__EFMigrationsHistory`, the `zbauth` schema present (inbound API keys, Auth `ApiKeys.SqlServer`), and the Secrets SqlServer store tables present. 4. First backup: `COPY_ONLY, CHECKSUM, COMPRESSION` + `VERIFYONLY`. +5. Insert the `_PROD` role rows by SQL (§3A.3). Gate: the rows read back; ids 1–5 (`SCADA-*`) untouched. ### Phase 4: MxAccessGateway ×4 @@ -149,9 +158,9 @@ the connection to the local PAR gateways as a normal connection edit and instanc Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the masked env before the real run. -- Environment `Production`, which **requires login on**. LDAPS to NAM (nam1/nam2 + `FallbackServers`), the A1P - bind, and GroupToRole for `_PROD` groups only: `APP_ADMIN_PROD`→Administrator, `APP_IDE_PROD`→Viewer, - `APP_IDE_RO_PROD`→Viewer (`ldap_mapping.md` §4.1). +- Environment `Production`, which **requires login on**. The LDAP env block and `GroupToRole` from §3A.3 go into the + `-EnvFile`; delete `GwAdmin`/`GwReader` from the base `appsettings.json` before the first start. +- Install **without `-Start`**, seed `nam-ldap-svc-password` (§3A.4), review the masked env, then start. - **Fresh prod secrets on each box:** `ZB_SECRETS_MASTER_KEY`, `MxGateway__ApiKeyPepper`. Nothing is copied from QA. Choose one of two patterns and record the choice: (a) **Recommended:** the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so @@ -160,7 +169,7 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the - Galaxy SQL and the alarm subscription per Q2. `SqlitePath` values are absolute and outside the content root. - Mint client keys: `scadabridge-site-prod` on the PAR gateways, plus whatever Q4 decides for the app boxes. Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else. -- Gates: dashboard login with a NAM `_PROD` user succeeds; a user with zero roles is refused; with the client +- Gates: §3A.5 (dashboard login as `dohertj2` → Administrator; bogus refused; no service-bind line); with the client token, `galaxy-test-connection` returns `ok:true`, and a session open/close works on each box; a bogus key gets a typed auth error; a worker session reports ready. @@ -172,9 +181,10 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the - the ConfigDb connection string; `Secrets:Store:Provider=SqlServer` (one store shared by both centrals); - a **shared `JwtSigningKey`** (≥ 32 random characters), shared ASP.NET Data Protection keys, and an `ApiKeyPepper`; - - NAM LDAPS with `AllowInsecure=false` and the A1P bind; - - GroupToRole for `_PROD` only, including `APP_DEPLOY_PROD`→Deployer (`Administrator` does not include - Deployer); + - NAM LDAPS: the §3A.3 block in each node's `appsettings.Central.json`, no `Auth` block, no LDAP env entry, + `RequireHttpsCookie` per D-P7; the secret seeded once into the shared store before the first start (§3A.4); + - roles come from the `LdapGroupMappings` rows inserted in Phase 3 (`_PROD` only, `APP_DEPLOY_PROD`→Deployer or + the D-P6 fallback; `Administrator` does not include Deployer); - downing `leader-auto-down` (the shipped default); OTLP to loopback if Phase 2 installed the collector. Start from the QA Central env **key names** (names only, never values) plus QA's appsettings shape. Write a per-node diff table before anything starts. @@ -184,8 +194,10 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the boundary the runbook describes). - Start order: vp03, wait for ready 200, then vp04. - Gates: ready 200 on both; active 200 on one node and 503 on the other; `required-singletons` healthy; - `bundle-signing-policy` healthy; a NAM login works in the UI and in the CLI (`--url`, A3P or a named - deployer); a zero-role user is refused at `/auth/login`, `/auth/token` and `/management`. + `bundle-signing-policy` healthy; §3A.5 on the active central: `dohertj2` in the UI, at `/auth/token` (role claims) + and in the CLI (`--url`, `--password-file`), bogus refused, no service-bind line; then the same login against the + other central after a failover drill in Phase 8. A zero-role user is refused at `/auth/login`, `/auth/token` and + `/management` where §3A.5 item 4 has an account for it. ### Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04) @@ -227,6 +239,128 @@ Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the - Overview: decide whether prod gets its own Overview, or a `Prod` registry on an existing box. It is not in scope unless the owner asks. +## 3A. LDAP settings (NAM AD) for PROD + +Login is on from the first start (D-P3). This section is the PROD form of the family cutover +([`2026-11-15-nam-ldap-cutover.md`](2026-11-15-nam-ldap-cutover.md)), which already ran on DEV and QA with these +exact builds (records [`docs/deploy/2026-11-19-vd03-nam-ldap/`](../deploy/2026-11-19-vd03-nam-ldap/README.md) and +[`docs/deploy/2026-11-19-qa-nam-ldap/`](../deploy/2026-11-19-qa-nam-ldap/README.md)). Differences from QA: PROD is +greenfield, so there is no GLAuth config to replace, and the settings are written at install time, before the first start. + +### 3A.1 Decisions + +| # | Decision | Recommendation | +|---|---|---| +| D-P5 | **Bind account.** PROD binds as **A1P `svc_zbmomww_prod`** (T6; no lockout, read-only). DEV and QA run on the interim `svc_igntadmin`, which belongs to the Ignition team. | **Wait for A1P.** Using `svc_igntadmin` in PROD means one Ignition-owned password (and one lockout) spans every tier and the Ignition PROD stack. Only with an explicit owner go **and** the Ignition team's OK; then the A1P swap is a re-seed of `nam-ldap-svc-password` in every PROD store plus a rolling restart. | +| D-P6 | **ScadaBridge Deployer.** `UG_WW_APP_DEPLOY_PROD` (G3, T6). | **Wait for G3.** Fallback, as on DEV/QA: `UG_WW_HIST_ADMIN_PROD`→Deployer. Note it is **not** the same membership as `APP_ADMIN_PROD` in PROD (4 vs 5 members on 2026-11-19: `millerh5` is admin but not hist-admin), so with the fallback that admin cannot deploy. | +| D-P7 | **Cookies over plain HTTP.** Each app's auth cookie must be `Secure` only when the endpoint is HTTPS. | If Q7 delivers certs: HTTPS endpoints and `RequireHttpsCookie=true`. Without TLS: `RequireHttpsCookie=false` on mxgw and ScadaBridge (as on QA), recorded as a known gap. | + +### 3A.2 Directory facts (2026-11-19 dump; re-verify in Phase 0) + +| Group | Members | ScadaBridge | MxAccessGateway | +|---|---|---|---| +| `UG_WW_APP_ADMIN_PROD` | 5 (chamalas, millerh5, marting4, dohertj2, Luis Bastidas) | Administrator | Administrator | +| `UG_WW_APP_IDE_PROD` | 4 (chamalas, marting4, dohertj2, Luis Bastidas) | Designer | Viewer | +| `UG_WW_APP_DEPLOY_PROD` | **does not exist yet** (G3) | Deployer | — | +| `UG_WW_APP_IDE_RO_PROD` | 1 (oswaltk) | Viewer | Viewer | +| `UG_WW_BTMGR_RO_PROD` | 4 (benjamd2, marting4, oswaltk, Luis Bastidas) | Viewer | — | +| `UG_WW_HIST_ADMIN_PROD` | 4 | (D-P6 fallback only) | — | + +`UG_WW_BTMGR_{BE,MC}_PROD` and `UG_WW_CITRIX_PROD` are **never** mapped here (§4.2 rows wait for later). Each PROD +deployment lists **only `_PROD` groups**; a `_DEV`/`_QA` name in a PROD config is a defect. The T6 membership review +must finish before the mapping goes live: `BTMGR_RO_PROD` gives 4 people ScadaBridge Viewer. + +**Test identity:** `NAM\dohertj2` is in `APP_ADMIN_PROD` and `APP_IDE_PROD`, so it is the PROD login probe (expected +roles: ScadaBridge Administrator+Designer, plus Deployer once G3 or D-P6 lands; mxgw Administrator). Its password is in +the Keychain as `winrm-cli/vq03`. `adm_dohertj2` is the box-admin account for winrm/SMB only, never a login probe. + +### 3A.3 The block (identical on every app; only the key prefix differs) + +| Setting | Value | +|---|---| +| `Enabled` | `true` | +| `Server` / `Port` / `Transport` | `uswap-vdc-nam1.nam.zimmer.com` / `636` / `Ldaps` | +| `FallbackServers[0]` | `uswap-vdc-nam2.nam.zimmer.com:636` | +| `AllowInsecure` | `false` (needs `zimmer-rootca` in `LocalMachine\Root`; Phase 2 gate) | +| `SearchBase` | `DC=nam,DC=zimmer,DC=com` | +| `ServiceAccountDn` | `svc_zbmomww_prod@nam.zimmer.com` (A1P; D-P5) | +| `ServiceAccountPassword` | `${secret:nam-ldap-svc-password}`, never a literal | +| `UserNameAttribute` / `DisplayNameAttribute` / `GroupAttribute` | `sAMAccountName` / `name` / `memberOf` | +| `BindMode` | `ServiceAccount` | + +**MxAccessGateway (×4): NSSM env**, through `Install-MxGatewayService.ps1 -EnvFile` (the QA pattern; the upgrade script +never writes the env, so it survives every redeploy): + +``` +MxGateway__Ldap__Enabled=true +MxGateway__Ldap__Server=uswap-vdc-nam1.nam.zimmer.com +MxGateway__Ldap__Port=636 +MxGateway__Ldap__Transport=Ldaps +MxGateway__Ldap__AllowInsecure=false +MxGateway__Ldap__FallbackServers__0=uswap-vdc-nam2.nam.zimmer.com:636 +MxGateway__Ldap__SearchBase=DC=nam,DC=zimmer,DC=com +MxGateway__Ldap__ServiceAccountDn=svc_zbmomww_prod@nam.zimmer.com +MxGateway__Ldap__ServiceAccountPassword=${secret:nam-ldap-svc-password} +MxGateway__Ldap__UserNameAttribute=sAMAccountName +MxGateway__Ldap__DisplayNameAttribute=name +MxGateway__Ldap__GroupAttribute=memberOf +MxGateway__Ldap__BindMode=ServiceAccount +MxGateway__Dashboard__GroupToRole__UG_WW_APP_ADMIN_PROD=Administrator +MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_PROD=Viewer +MxGateway__Dashboard__GroupToRole__UG_WW_APP_IDE_RO_PROD=Viewer +MxGateway__Dashboard__RequireHttpsCookie= +``` + +Plus, in the base `app\appsettings.json`, **delete `MxGateway:Dashboard:GroupToRole` (`GwAdmin`/`GwReader`)** before the +first start: configuration merges the file map with the env map, so the GLAuth names would otherwise stay mapped. The +upgrade script carries the edited file forward on later upgrades. + +**ScadaBridge Central (×2): `app\appsettings.Central.json`**, section `ScadaBridge:Security:Ldap` (the QA shape; each +node's file is written separately, never copied over the other), with no `Auth` block (`DisableLogin` / +`AllowOutsideDevelopment` absent) and `ScadaBridge:Security:RequireHttpsCookie` per D-P7. **No +`ScadaBridge__Security__Ldap__*` env entry**: an env password would override the `${secret:}` reference. Sites have no +login and get no LDAP settings. + +**ScadaBridge role rows**, in `dbo.LdapGroupMappings` (the central database, not config). With login on from the first +start nobody can sign in until they exist, so insert them **by SQL after Phase 3** with the idempotent script +`docs/deploy/2026-11-19-vd03-nam-ldap/sb-role-mappings.ps1` retargeted to these rows: +`UG_WW_APP_ADMIN_PROD`→Administrator, `UG_WW_APP_IDE_PROD`→Designer, `UG_WW_APP_IDE_RO_PROD`→Viewer, +`UG_WW_BTMGR_RO_PROD`→Viewer, `UG_WW_APP_DEPLOY_PROD`→Deployer (or the D-P6 fallback). Later changes go through the CLI +(audited). The migration seeds five `SCADA-*` rows (ids 1–5); they grant nothing unless a NAM group with that exact +name exists, which Phase 0 checks (`LdapProbe -g 'SCADA-*'` returns none). + +### 3A.4 The secret, before the first start + +Each app's pre-host expander fails closed on a missing `${secret:}`, so `nam-ldap-svc-password` must be in the store +before the service is started for the first time: + +| Store | Where | How | +|---|---|---| +| mxgw ×4 | each box's `Secrets__SqlitePath` (absolute, outside the content root), KEK per that service's env | `Install-MxGatewayService.ps1` **without `-Start`**, then `seed-secret.ps1` (the QA one; it reads the store path and KEK from the service env), then start | +| ScadaBridge centrals | the shared SQL Server store (`zbsecrets`, `Secrets:Store:Provider=SqlServer`), **once** | the SQL-aware Secrets CLI build (the headless 0.6.6 CLI writes SQLite only; QA finding 2). Both centrals must hold the **same KEK file** (check the hash on both) | + +The value goes Keychain → 0600 temp file → SMB → ACL'd file → CLI stdin → deleted; read back and compared on the box. +It is never on argv, in chat or in a log. Use the **Secrets CLI 0.6.6** (stdin); 0.6.2 stores "-" literally. + +### 3A.5 Verification and stop rule (after each app's first start) + +1. Anonymous `GET /` → 302 `/login`. +2. A bogus user is refused, with no `system-side` / service-bind line in the log. +3. `dohertj2` signs in with the expected roles (ScadaBridge: cookie, `/auth/token` claims, and the CLI over HTTP Basic + against the active central; mxgw: dashboard). +4. Zero-role refusal: needs a NAM account the operator controls that is in no `_PROD` group. If there is none, rely on + the DEV/QA proofs of #144/#145 and say so in the record. +5. Logs: the app's own log or the `ZbOtelCollector` export if Phase 2 installed it (HG-style empty app logs happened on DEV). + +**Stop at the first service-account bind failure** (the A1P account is no-lockout, but stop anyway and check with IT), +not the second. No LDAP DC-failover drill (blocking nam1 or pointing at a closed port) and no ad-hoc firewall rules for +testing (owner, 2026-11-19, for the DEV/QA cutover; carried over here). The §4 inbound rules are a separate, planned change. + +### 3A.6 Rollback + +PROD is greenfield until Phase 7 step 5, so an LDAP problem before then means: stop the service, fix the env, file or +rows, start again. There is no GLAuth to fall back to and none is installed. + ## 4. Port and firewall matrix (to request in Phase 0) | From | To | Port | Purpose | @@ -264,6 +398,8 @@ the existing site gRPC listener. | The central↔site boundary (`8083`) is blocked or filtered in one direction only. QA never exercised it, because it was co-located | Phase 0 firewall ticket; Phase 6 round-trip gate in **both** directions | | One machine driven by two stacks (Q9) | Undeploy from QA before the prod deploy | | `_PROD` groups grant wider access than intended | T6 membership review before mapping | -| A service-account lockout breaks every app | A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule) | +| A service-account lockout breaks every app | A1P is no-lockout; stop at the **first** service-account bind failure (§3A.5). Using the Ignition-owned `svc_igntadmin` in PROD (D-P5 fallback) would tie PROD to Ignition's password | +| `${secret:nam-ldap-svc-password}` missing at first start (the expander fails closed and the service will not boot) | Seed before the first start (§3A.4); install mxgw without `-Start` | +| A `_DEV`/`_QA` group name in a PROD config grants lower-tier admins PROD roles | §3A.2 rule; review the env file and the role rows against §3A.3 before the start | | The mxgw upgrade script aborts on a slow start | Phase 1 step 1 fix | | The security software flags remote exec | Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert |