From 91d35edb9c1fea5faef13ac500974bd1d53dc09b Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 19 Nov 2026 14:44:26 -0500 Subject: [PATCH] docs(plan): PROD install of ScadaBridge + mxaccessgw (app-vp03/04 central, par-vp03/04 site) Plan only, nothing executed. Records owner decisions (mxgw on all four boxes, current main via QA, NAM AD LDAPS day one, signed QA bundle), the SMB-verified box facts (admin account, subnets, .NET, MXAccess 6000.0045.2657.1; PAR boxes wait on the WinPlatform deploy), phases 0-8, port matrix, rollback and open questions. --- ...026-11-19-prod-scadabridge-mxgw-install.md | 269 ++++++++++++++++++ 1 file changed, 269 insertions(+) create mode 100644 docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md diff --git a/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md new file mode 100644 index 0000000..1a56889 --- /dev/null +++ b/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md @@ -0,0 +1,269 @@ +# PROD install: ScadaBridge + MxAccessGateway (plan, 2026-11-19) + +**State: PLAN ONLY. Nothing has been executed, and no prod box has been touched.** This is the first production +tier for the family. Until now `deployment.md` said "no production box has been identified". + +**The DEPLOY HOLD still applies to QA and PROD** (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every +phase below that touches QA or a prod box needs its own explicit owner go. + +## 1. Target topology + +| Host | Role | ScadaBridge | MxAccessGateway | +|---|---|---|---| +| `wonder-app-vp03` | Central A | `ScadaBridge-Central` (UI/Management/Inbound API `:8085`, Akka `:8081`, central gRPC `:8083`) | `MxAccessGw` `:5120` gRPC / `:5130` dashboard | +| `wonder-app-vp04` | Central B | `ScadaBridge-Central` (same ports) | `MxAccessGw` (same ports) | +| `WONDER-PAR-VP03` | Site node A | `ScadaBridge-Site` (Akka `:8082`, gRPC `:8083`, health `:8084`) | `MxAccessGw` (same ports) | +| `WONDER-PAR-VP04` | Site node B | `ScadaBridge-Site` (same ports) | `MxAccessGw` (same ports) | + +**This differs from QA.** On QA, vq03/vq04 each run a central node *and* a site node. In prod the central pair +and the site pair are on different boxes, and possibly on different network segments (`app` vs `PAR`). The +central↔site gRPC control plane (`:8083`, both directions, PSK-authenticated) therefore crosses a network +boundary for the first time. It is the main new risk in this install. + +**Decisions recorded (owner, 2026-11-19):** +- **D-P1 mxgw on all four boxes.** Each box needs AVEVA System Platform with MXAccess registered + (`LMXProxy.LMXProxyServer`). The site pair's MxGateway connections use the PAR gateways. The purpose of the + central-box gateways has to be confirmed (see Q4). +- **D-P2 builds: current `main`, promoted through QA first.** ScadaBridge `2e0ce81e` (includes #144/#145, + which refuse zero-role logins) and mxgw `e74918cc`. Prod gets the **same zips, byte-identical** to the ones + that soaked on QA. +- **D-P3 NAM AD over LDAPS from day one.** This uses `_PROD` groups only and separate PROD accounts. There is + no GLAuth in prod. +- **D-P4 content: a signed config bundle from QA**, limited to the machines selected for prod. Prod + connections are re-pointed at the prod gateways, and prod mints its own inbound API keys. + +## 2. Open questions (each blocks the phase noted) + +| # | Question | Blocks | +|---|---|---| +| Q1 | Which SQL Server hosts the prod `ScadaBridge` ConfigDb? Who creates the DB and the `scadabridge_svc` login? | Phase 3 | +| Q2 | Where is the prod Galaxy Repository (`ZB` DB) and its GR node, and which alarm area do the gateways subscribe to (`\\\Galaxy!`)? | Phase 4 | +| Q3 | **Partly answered over SMB (2026-11-19, read-only, table below). ⚠ The PAR boxes have no MXAccess.** `app-vp03/04` have a full Application Server install: `ArchestrA\Framework\Bin` holds 562 entries, including `LmxProxy.dll`. `PAR-VP03/04` hold only 49 entries there, with no `LmxProxy.dll`; they carry OI Server, Historian Client and the common components. Reference: prod runtime node `wonder-bd5-vp03` has MXAccess with `aaEngine.exe` and `LmxProxy.dll` 6000.0045.2657.1 (131 entries), byte-identical to vp03 and to QA vq03; that is the level the PAR boxes need. **Owner, 2026-11-19: WinPlatform is not deployed to the PAR boxes yet. Deploying it from the prod Galaxy installs MXAccess there.** So mxgw on PAR is gated on that deploy (an AVEVA/Galaxy step, outside this plan). Gate check over SMB: `C$\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll` is present at 6000.0045.2657.1 (SHA prefix `bd75a2fde222`). Until then, **mxgw cannot run on the PAR boxes**: it needs a WinPlatform deployed to the box. The alternative is to point the site's MxGateway connections at the app-box gateways (see Q4). Still open: NSSM (none found under `Program Files`) and free disk. | Phase 2, Phase 4 | +| Q4 | What consumes the gateways on `app-vp03/04`? ScadaBridge centrals don't talk to mxgw. If the answer is "nothing yet" (for example, a future OtOpcUa), install them but mint no client keys. | Phase 4 | +| Q5 | Network: DNS resolved from the Mac on 2026-11-19 (table below). The centrals and the sites are on **different subnets** (`10.220.153.x` vs `10.201.75.x`), so a firewall between them is likely. Find out whether one exists. See the port matrix in §4. | Phase 0 | +| Q6 | Remote deploy from the Mac: the account is settled (below). Still open: whether WinRM 5985 and SMB 445 are open from the Mac to the four boxes, and whether a prod change needs a corporate change ticket. The security software tripped on WinRM before (2026-09-11). | Every phase on a box | + +**Hosts (DNS from the Mac, 2026-11-19):** + +| FQDN | IP | +|---|---| +| `wonder-app-vp03.zmr.zimmer.com` | 10.220.153.205 | +| `wonder-app-vp04.zmr.zimmer.com` | 10.220.153.210 | +| `wonder-par-vp03.zmr.zimmer.com` | 10.201.75.86 | +| `wonder-par-vp04.zmr.zimmer.com` | 10.201.75.87 | + +QA's vq03/vq04 sit on 10.220.158.x. + +**Remote-access account:** QA and PROD boxes outside the wonder DEV/QA set need the admin account +**`NAM\adm_dohertj2`**. `NAM\dohertj2` gets `STATUS_ACCESS_DENIED` on the `C$` share of the DNC QA/PROD +boxes. That account is already in the winrm Keychain as profiles `dnc-vq01` (`dnc-app-vq01`) and `dnc-vp01` +(`dnc-app-vp01`), both NTLM over HTTP 5985. The prod profiles are `vp03`, `vp04`, `parvp03` and `parvp04`, as +`NAM\adm_dohertj2`, with the password copied Keychain→Keychain from `winrm-cli/dnc-vp01` (the same method used +for `sqlvd03`). Whether `adm_dohertj2` is a local admin on the wonder prod boxes is **unverified**. + +**SMB check (2026-11-19, `winrm smb ls` only, no remote exec):** +- `NAM\adm_dohertj2` opens `C$` and `E$` on all four boxes, so it has admin-share rights. None of them has a + `D$` share. +- No `E:\ApiInstall` exists yet on any box, so this is greenfield. Each box has `E:\2023R2Install` (AVEVA 2023 R2 + media) and `E:\Hotfixes`. `vp03`'s `E:` also holds `ProdPromotions`, `ServiceIntegrations` and `Backups`. +- .NET: only 6.0.25 and 8.0.0, so the builds must be self-contained. +- `Program Files\Microsoft SQL Server` is present on vp03 and parvp03. Whether those instances are what Q1/Q2 need + is unverified. + +**DELMIA prod is `dnc-app-vp01`** (10.201.207.106). For Q8, that box is the prod Delmia caller of +ScadaBridge's `DelmiaRecipeDownload`. Prod runs an older Downloader than QA (`delmia` project memory), so check +the prod integration's shape before you assume it matches QA's. +| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. | Phases 4–5 | +| Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the `SiteId`? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 | +| Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 | + +## 3. Phases + +### Phase 0: approvals and external prerequisites (owner + IT, before any box work) + +1. **Ticket T6** (`nam_ad_requests.md` §6), filed with IT: + - A1P `svc_zbmomww_prod`: the LDAP lookup account, no lockout, read-only; + - A3P `svc_zbmomww_auto_prod`, if the owner allows PROD automation (plan D3); + - G3 `UG_WW_APP_DEPLOY_PROD`; + - a **membership review** of `UG_WW_APP_ADMIN_PROD`, `APP_IDE_PROD`, `APP_IDE_RO_PROD` and `BTMGR_RO_PROD`. + These groups were built for System Platform, so mapping them grants their members app roles in prod. +2. Extend scadaproj#58 for A1P (comment). File a `cross-repo` umbrella issue in scadaproj, plus a `type:deploy-gate` + issue in ScadaBridge and in mxaccessgw, cross-linked. +3. Firewall requests, from the §4 matrix. +4. SQL (Q1): an empty `ScadaBridge` DB, the `scadabridge_svc` login as `db_owner`, and a backup target. +5. Certificates (Q7). +6. Access (Q6). If the Mac path is allowed, add `winrm` profiles `vp03`, `vp04`, `parvp03` and `parvp04` with + Keychain credentials, then run an SMB-only reachability check (`winrm smb ls`). **No remote exec before the + owner's go.** +7. Owner go for QA promotion (Phase 1). + +### Phase 1: candidate builds proven on QA + +1. **Fix mxgw `scripts/deploy/Upgrade-MxGatewayService.ps1` first.** `nssm start` returning `SERVICE_START_PENDING` + makes it abort and auto-roll-back. The fix is the `try/catch` and the `Wait-State` already used on vd03 + (memory note). Merge it to mxgw `main`. If it lands after `e74918cc`, the candidate SHA moves, and the build + is re-proven on DEV first. +2. **Build once**, then reuse the outputs for QA and PROD: + - ScadaBridge Host and CLI: `dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`; + - mxgw Server: self-contained win-x64; + - mxgw x86 Worker on **windev**, at the **same SHA** (`-p:Platform=x86 -p:SourceRevisionId=`). The upgrade + script refuses mixed SHAs. + Record a SHA-256 for every zip. +3. Deploy to QA with the existing 061432c8-style pack. ScadaBridge needs **no migration** (`061432c8..2e0ce81e` + has no migration change; the schema stays at 65). mxgw rolls one box at a time. +4. **Soak on QA for at least 3 working days.** The gates are the QA record's set: ready 200 ×4, one active per + pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative + control). +5. Optionally, do QA's NAM cutover (plan Phases A–7) before prod. **Recommended.** QA is then the first + environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first. + +### Phase 2: box prep (all four, no service started) + +- Layout as on QA: `E:\ApiInstall\{ScadaBridge,MxAccessGateway}\{app,cli|worker,data,logs}`, with staging in + `E:\psrem-stage\-\`. +- NSSM for mxgw. ScadaBridge services are created with `sc.exe` (installation guide), env in the service key + `Environment` (REG_MULTI_SZ). +- KEK folders `C:\ProgramData\ZB\kek\` set to `Administrators:F + SYSTEM:F`, with inheritance off. +- Inbound firewall rules on each box, per §4. +- Optional, and recommended for parity: the `ZbOtelCollector` loopback collector (`infra/otel-collector/wonder/`). +- Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN + that the config will use. + +### Phase 3: ConfigDb + +1. Generate a **full, non-idempotent** script from the empty DB to the last migration at the candidate SHA: + `dotnet ef migrations script 0 --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase`. QA learned + that a full-history `--idempotent` script fails at batch compile. +2. Review it, then run it with `sqlcmd -I -b` (QUOTED_IDENTIFIER, or the filtered indexes fail with Msg 1934) + from a box that has `sqlcmd`, using the service login. Alternatively, a DBA runs it. +3. Gate: 65 rows in `__EFMigrationsHistory`, the `zbauth` schema present (inbound API keys, Auth + `ApiKeys.SqlServer`), and the Secrets SqlServer store tables present. +4. First backup: `COPY_ONLY, CHECKSUM, COMPRESSION` + `VERIFYONLY`. + +### Phase 4: MxAccessGateway ×4 + +**Order:** install on `app-vp03/04` first, since MXAccess is already there. Install on `PAR-VP03/04` only after the +WinPlatform deploy puts MXAccess on them (Q3 gate). Until the PAR gateways exist, the site's MxGateway connection can +point at the app-box gateways. That needs `5120` open from `10.201.75.x` to `10.220.153.x`. Afterwards, re-point +the connection to the local PAR gateways as a normal connection edit and instance redeploy. + +Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the masked env before the real run. + +- Environment `Production`, which **requires login on**. LDAPS to NAM (nam1/nam2 + `FallbackServers`), the A1P + bind, and GroupToRole for `_PROD` groups only: `APP_ADMIN_PROD`→Administrator, `APP_IDE_PROD`→Viewer, + `APP_IDE_RO_PROD`→Viewer (`ldap_mapping.md` §4.1). +- **Fresh prod secrets on each box:** `ZB_SECRETS_MASTER_KEY`, `MxGateway__ApiKeyPepper`. Nothing is copied from + QA. Choose one of two patterns and record the choice: + (a) **Recommended:** the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so + one client token works on both site gateways; + (b) independent stores, with a key per box. +- Galaxy SQL and the alarm subscription per Q2. `SqlitePath` values are absolute and outside the content root. +- Mint client keys: `scadabridge-site-prod` on the PAR gateways, plus whatever Q4 decides for the app boxes. + Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else. +- Gates: dashboard login with a NAM `_PROD` user succeeds; a user with zero roles is refused; with the client + token, `galaxy-test-connection` returns `ok:true`, and a session open/close works on each box; a bogus key + gets a typed auth error; a worker session reports ready. + +### Phase 5: ScadaBridge central pair (vp03, vp04) + +- Config, per node: + - `Node:Role=Central`, `NodeHostname` = the FQDN; + - seed nodes **self first** (StartupValidator enforces it, with the same spelling as `NodeHostname`); + - the ConfigDb connection string; `Secrets:Store:Provider=SqlServer` (one store shared by both centrals); + - a **shared `JwtSigningKey`** (≥ 32 random characters), shared ASP.NET Data Protection keys, and an + `ApiKeyPepper`; + - NAM LDAPS with `AllowInsecure=false` and the A1P bind; + - GroupToRole for `_PROD` only, including `APP_DEPLOY_PROD`→Deployer (`Administrator` does not include + Deployer); + - downing `leader-auto-down` (the shipped default); OTLP to loopback if Phase 2 installed the collector. + Start from the QA Central env **key names** (names only, never values) plus QA's appsettings shape. Write a + per-node diff table before anything starts. +- Recovery, as on QA: restart 5 s / 5 s / 10 s, reset 86 400, `failureflag 1` (exit 75 recovers). +- Bundle signing: generate the prod signing key (`docs/operations/bundle-signing-runbook.md`), add the QA + exporter's fingerprint to the trust list, and turn `RequireSignedBundles` **on** (this is the production + boundary the runbook describes). +- Start order: vp03, wait for ready 200, then vp04. +- Gates: ready 200 on both; active 200 on one node and 503 on the other; `required-singletons` healthy; + `bundle-signing-policy` healthy; a NAM login works in the UI and in the CLI (`--url`, A3P or a named + deployer); a zero-role user is refused at `/auth/login`, `/auth/token` and `/management`. + +### Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04) + +- On the central: create the `Site` entity (`SiteId` per Q8), with `GrpcNodeAAddress`/`GrpcNodeBAddress` set to + `http://:8083`. Write `SB-GRPC-PSK-` through `/admin/secrets`. +- Site config: + - `Role=Site`, `SiteId`, seed nodes self first; + - `Communication:CentralGrpcEndpoints` = both centrals' `:8083`; `GrpcPsk` = `${secret:SB-GRPC-PSK-}`; + - `LocalDb:Replication:ApiKey` byte-identical on both nodes; `PeerAddress` on **one** node only; + - absolute paths for `AuditLog:SiteWriter:DatabasePath` and the LocalDb path (QA once found the site audit DB + in System32). +- **Start the pair together**, because a site pair cannot be rolled one node at a time. +- Gates: ready 200 ×2; one node active; site online on the central health dashboard; LocalDb + `replicationConnected=true` with backlog 0; a central→site command round-trip; the site's MxGateway connection + to PAR-VP03 is connected. + +### Phase 7: content and first machines + +1. On QA, export a **signed** bundle containing only the selected templates, connections, inbound methods and + db-connections (Q8). +2. Import it on prod. Check that no key and no QA hostname came along: re-point each MxGateway connection to + `PAR-VP03:5120` primary and `PAR-VP04` backup, the BTDB db-connection to prod, and any OPC UA connection. +3. Mint inbound API keys on prod for MES and Delmia (`inbound-api-key-reissue.md`). Hand them over out of band. +4. Undeploy the matching machines from QA, or confirm Q9 another way. Then deploy **one machine first**, during + an idle window: verify attributes Good, alarms present, and one read-only inbound call (a `{}` body returns + 400 `VALIDATION_FAILED`, which proves the key works). Then deploy the rest in batches. +5. Point MES and Delmia at prod only after step 4 passes. **This is the point of no return for consumers.** + Before it, rollback = stop the prod services. + +### Phase 8: drills and handover + +- Failover drills on idle machines only, copying the QA #116 procedure: on each pair, a kill drill and a + stop-and-hold drill. Expected takeover is about 8 s for a kill and about 27 s for stop-and-hold. +- Records: + - `ScadaBridge/docs/deployment/deployment-records/-wonder-app-vp03-vp04-par-vp03-vp04-.md`; + - mxgw `docs/deploy/-prod-.md`. +- Docs to update: `deployment.md` (§1 inventory, §2.1 hosts, a new §7.2 PROD port map, §9), the CLAUDE.md index + (on `main`, pushed), and the memory notes `wonder-environment-tiers` and `nam-ldap-cutover-handoff`. +- Overview: decide whether prod gets its own Overview, or a `Prod` registry on an existing box. It is not in + scope unless the owner asks. + +## 4. Port and firewall matrix (to request in Phase 0) + +| From | To | Port | Purpose | +|---|---|---|---| +| vp03 ↔ vp04 | each other | 8081 TCP | Central Akka remoting | +| PAR-VP03 ↔ PAR-VP04 | each other | 8082 TCP | Site Akka remoting | +| PAR-VP03 ↔ PAR-VP04 | each other | 8083 TCP (h2c) | LocalDb sync stream (rides the site gRPC listener) | +| PAR-VP0x | vp03, vp04 | 8083 TCP (h2c) | Site → `CentralControlService` | +| vp03, vp04 | PAR-VP0x | 8083 TCP (h2c) | Central → `SiteCommandService`, streaming, audit pull | +| PAR-VP0x (Site) | PAR-VP03/04 | 5120 TCP | ScadaBridge MxGateway adapter | +| vp03, vp04 | prod SQL | 1433 TCP | ConfigDb, Secrets, zbauth | +| all four | GR node SQL | 1433 TCP | mxgw Galaxy browse (Q2) | +| vp03, vp04, all mxgw | nam1/nam2 | 636 TCP | NAM LDAPS | +| users, MES, Delmia | vp03, vp04 | 8085 (or 443 with TLS) | Central UI, Management, Inbound API | +| admins | all four | 5130 | mxgw dashboard | +| central and site | SMTP or EWS | 587 / 443 | Notifications, if in scope | +| Mac (deploy) | all four | 5985, 445 | WinRM deploy scope + SMB (only if Q6 allows) | + +The LocalDb sync row is verified: the rig's `site-a-node-a` uses `PeerAddress: http://scadabridge-site-a-b:8083`, +the existing site gRPC listener. + +## 5. Rollback + +- **Before Phase 7 step 5**, prod is greenfield. Rollback = stop and disable the services. QA keeps serving + every consumer. +- **After consumers are cut over**, rollback = point MES and Delmia back at QA, and redeploy the moved machines on + QA. The `.bak` folders and DB backup follow the same pattern as QA (binaries + `COPY_ONLY` backup per window). +- Every later prod upgrade uses the QA pack pattern: stop both sites, stop both centrals, migrate, swap, start + the centrals, start the sites. + +## 6. Risks + +| Risk | Mitigation | +|---|---| +| The central↔site boundary (`8083`) is blocked or filtered in one direction only. QA never exercised it, because it was co-located | Phase 0 firewall ticket; Phase 6 round-trip gate in **both** directions | +| One machine driven by two stacks (Q9) | Undeploy from QA before the prod deploy | +| `_PROD` groups grant wider access than intended | T6 membership review before mapping | +| A service-account lockout breaks every app | A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule) | +| The mxgw upgrade script aborts on a slow start | Phase 1 step 1 fix | +| The security software flags remote exec | Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert |