docs(plan): PROD install of ScadaBridge + mxaccessgw (app-vp03/04 central, par-vp03/04 site)

Plan only, nothing executed. Records owner decisions (mxgw on all four boxes,
current main via QA, NAM AD LDAPS day one, signed QA bundle), the SMB-verified
box facts (admin account, subnets, .NET, MXAccess 6000.0045.2657.1; PAR boxes
wait on the WinPlatform deploy), phases 0-8, port matrix, rollback and open
questions.
This commit is contained in:
Joseph Doherty
2026-11-19 14:44:26 -05:00
parent b9d13e8856
commit 91d35edb9c
@@ -0,0 +1,269 @@
# PROD install: ScadaBridge + MxAccessGateway (plan, 2026-11-19)
**State: PLAN ONLY. Nothing has been executed, and no prod box has been touched.** This is the first production
tier for the family. Until now `deployment.md` said "no production box has been identified".
**The DEPLOY HOLD still applies to QA and PROD** (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every
phase below that touches QA or a prod box needs its own explicit owner go.
## 1. Target topology
| Host | Role | ScadaBridge | MxAccessGateway |
|---|---|---|---|
| `wonder-app-vp03` | Central A | `ScadaBridge-Central` (UI/Management/Inbound API `:8085`, Akka `:8081`, central gRPC `:8083`) | `MxAccessGw` `:5120` gRPC / `:5130` dashboard |
| `wonder-app-vp04` | Central B | `ScadaBridge-Central` (same ports) | `MxAccessGw` (same ports) |
| `WONDER-PAR-VP03` | Site node A | `ScadaBridge-Site` (Akka `:8082`, gRPC `:8083`, health `:8084`) | `MxAccessGw` (same ports) |
| `WONDER-PAR-VP04` | Site node B | `ScadaBridge-Site` (same ports) | `MxAccessGw` (same ports) |
**This differs from QA.** On QA, vq03/vq04 each run a central node *and* a site node. In prod the central pair
and the site pair are on different boxes, and possibly on different network segments (`app` vs `PAR`). The
central↔site gRPC control plane (`:8083`, both directions, PSK-authenticated) therefore crosses a network
boundary for the first time. It is the main new risk in this install.
**Decisions recorded (owner, 2026-11-19):**
- **D-P1 mxgw on all four boxes.** Each box needs AVEVA System Platform with MXAccess registered
(`LMXProxy.LMXProxyServer`). The site pair's MxGateway connections use the PAR gateways. The purpose of the
central-box gateways has to be confirmed (see Q4).
- **D-P2 builds: current `main`, promoted through QA first.** ScadaBridge `2e0ce81e` (includes #144/#145,
which refuse zero-role logins) and mxgw `e74918cc`. Prod gets the **same zips, byte-identical** to the ones
that soaked on QA.
- **D-P3 NAM AD over LDAPS from day one.** This uses `_PROD` groups only and separate PROD accounts. There is
no GLAuth in prod.
- **D-P4 content: a signed config bundle from QA**, limited to the machines selected for prod. Prod
connections are re-pointed at the prod gateways, and prod mints its own inbound API keys.
## 2. Open questions (each blocks the phase noted)
| # | Question | Blocks |
|---|---|---|
| Q1 | Which SQL Server hosts the prod `ScadaBridge` ConfigDb? Who creates the DB and the `scadabridge_svc` login? | Phase 3 |
| Q2 | Where is the prod Galaxy Repository (`ZB` DB) and its GR node, and which alarm area do the gateways subscribe to (`\\<GR node>\Galaxy!<area>`)? | Phase 4 |
| Q3 | **Partly answered over SMB (2026-11-19, read-only, table below). ⚠ The PAR boxes have no MXAccess.** `app-vp03/04` have a full Application Server install: `ArchestrA\Framework\Bin` holds 562 entries, including `LmxProxy.dll`. `PAR-VP03/04` hold only 49 entries there, with no `LmxProxy.dll`; they carry OI Server, Historian Client and the common components. Reference: prod runtime node `wonder-bd5-vp03` has MXAccess with `aaEngine.exe` and `LmxProxy.dll` 6000.0045.2657.1 (131 entries), byte-identical to vp03 and to QA vq03; that is the level the PAR boxes need. **Owner, 2026-11-19: WinPlatform is not deployed to the PAR boxes yet. Deploying it from the prod Galaxy installs MXAccess there.** So mxgw on PAR is gated on that deploy (an AVEVA/Galaxy step, outside this plan). Gate check over SMB: `C$\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll` is present at 6000.0045.2657.1 (SHA prefix `bd75a2fde222`). Until then, **mxgw cannot run on the PAR boxes**: it needs a WinPlatform deployed to the box. The alternative is to point the site's MxGateway connections at the app-box gateways (see Q4). Still open: NSSM (none found under `Program Files`) and free disk. | Phase 2, Phase 4 |
| Q4 | What consumes the gateways on `app-vp03/04`? ScadaBridge centrals don't talk to mxgw. If the answer is "nothing yet" (for example, a future OtOpcUa), install them but mint no client keys. | Phase 4 |
| Q5 | Network: DNS resolved from the Mac on 2026-11-19 (table below). The centrals and the sites are on **different subnets** (`10.220.153.x` vs `10.201.75.x`), so a firewall between them is likely. Find out whether one exists. See the port matrix in §4. | Phase 0 |
| Q6 | Remote deploy from the Mac: the account is settled (below). Still open: whether WinRM 5985 and SMB 445 are open from the Mac to the four boxes, and whether a prod change needs a corporate change ticket. The security software tripped on WinRM before (2026-09-11). | Every phase on a box |
**Hosts (DNS from the Mac, 2026-11-19):**
| FQDN | IP |
|---|---|
| `wonder-app-vp03.zmr.zimmer.com` | 10.220.153.205 |
| `wonder-app-vp04.zmr.zimmer.com` | 10.220.153.210 |
| `wonder-par-vp03.zmr.zimmer.com` | 10.201.75.86 |
| `wonder-par-vp04.zmr.zimmer.com` | 10.201.75.87 |
QA's vq03/vq04 sit on 10.220.158.x.
**Remote-access account:** QA and PROD boxes outside the wonder DEV/QA set need the admin account
**`NAM\adm_dohertj2`**. `NAM\dohertj2` gets `STATUS_ACCESS_DENIED` on the `C$` share of the DNC QA/PROD
boxes. That account is already in the winrm Keychain as profiles `dnc-vq01` (`dnc-app-vq01`) and `dnc-vp01`
(`dnc-app-vp01`), both NTLM over HTTP 5985. The prod profiles are `vp03`, `vp04`, `parvp03` and `parvp04`, as
`NAM\adm_dohertj2`, with the password copied Keychain→Keychain from `winrm-cli/dnc-vp01` (the same method used
for `sqlvd03`). Whether `adm_dohertj2` is a local admin on the wonder prod boxes is **unverified**.
**SMB check (2026-11-19, `winrm smb ls` only, no remote exec):**
- `NAM\adm_dohertj2` opens `C$` and `E$` on all four boxes, so it has admin-share rights. None of them has a
`D$` share.
- No `E:\ApiInstall` exists yet on any box, so this is greenfield. Each box has `E:\2023R2Install` (AVEVA 2023 R2
media) and `E:\Hotfixes`. `vp03`'s `E:` also holds `ProdPromotions`, `ServiceIntegrations` and `Backups`.
- .NET: only 6.0.25 and 8.0.0, so the builds must be self-contained.
- `Program Files\Microsoft SQL Server` is present on vp03 and parvp03. Whether those instances are what Q1/Q2 need
is unverified.
**DELMIA prod is `dnc-app-vp01`** (10.201.207.106). For Q8, that box is the prod Delmia caller of
ScadaBridge's `DelmiaRecipeDownload`. Prod runs an older Downloader than QA (`delmia` project memory), so check
the prod integration's shape before you assume it matches QA's.
| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs `zimmer-rootca` in the machine store (cutover-plan D4), and `AllowInsecure=false` in prod. | Phases 4–5 |
| Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the `SiteId`? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 |
| Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 |
## 3. Phases
### Phase 0: approvals and external prerequisites (owner + IT, before any box work)
1. **Ticket T6** (`nam_ad_requests.md` §6), filed with IT:
- A1P `svc_zbmomww_prod`: the LDAP lookup account, no lockout, read-only;
- A3P `svc_zbmomww_auto_prod`, if the owner allows PROD automation (plan D3);
- G3 `UG_WW_APP_DEPLOY_PROD`;
- a **membership review** of `UG_WW_APP_ADMIN_PROD`, `APP_IDE_PROD`, `APP_IDE_RO_PROD` and `BTMGR_RO_PROD`.
These groups were built for System Platform, so mapping them grants their members app roles in prod.
2. Extend scadaproj#58 for A1P (comment). File a `cross-repo` umbrella issue in scadaproj, plus a `type:deploy-gate`
issue in ScadaBridge and in mxaccessgw, cross-linked.
3. Firewall requests, from the §4 matrix.
4. SQL (Q1): an empty `ScadaBridge` DB, the `scadabridge_svc` login as `db_owner`, and a backup target.
5. Certificates (Q7).
6. Access (Q6). If the Mac path is allowed, add `winrm` profiles `vp03`, `vp04`, `parvp03` and `parvp04` with
Keychain credentials, then run an SMB-only reachability check (`winrm smb ls`). **No remote exec before the
owner's go.**
7. Owner go for QA promotion (Phase 1).
### Phase 1: candidate builds proven on QA
1. **Fix mxgw `scripts/deploy/Upgrade-MxGatewayService.ps1` first.** `nssm start` returning `SERVICE_START_PENDING`
makes it abort and auto-roll-back. The fix is the `try/catch` and the `Wait-State` already used on vd03
(memory note). Merge it to mxgw `main`. If it lands after `e74918cc`, the candidate SHA moves, and the build
is re-proven on DEV first.
2. **Build once**, then reuse the outputs for QA and PROD:
- ScadaBridge Host and CLI: `dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false`;
- mxgw Server: self-contained win-x64;
- mxgw x86 Worker on **windev**, at the **same SHA** (`-p:Platform=x86 -p:SourceRevisionId=<sha>`). The upgrade
script refuses mixed SHAs.
Record a SHA-256 for every zip.
3. Deploy to QA with the existing 061432c8-style pack. ScadaBridge needs **no migration** (`061432c8..2e0ce81e`
has no migration change; the schema stays at 65). mxgw rolls one box at a time.
4. **Soak on QA for at least 3 working days.** The gates are the QA record's set: ready 200 ×4, one active per
pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative
control).
5. Optionally, do QA's NAM cutover (plan Phases A–7) before prod. **Recommended.** QA is then the first
environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first.
### Phase 2: box prep (all four, no service started)
- Layout as on QA: `E:\ApiInstall\{ScadaBridge,MxAccessGateway}\{app,cli|worker,data,logs}`, with staging in
`E:\psrem-stage\<app>-<sha>\`.
- NSSM for mxgw. ScadaBridge services are created with `sc.exe` (installation guide), env in the service key
`Environment` (REG_MULTI_SZ).
- KEK folders `C:\ProgramData\ZB\kek\` set to `Administrators:F + SYSTEM:F`, with inheritance off.
- Inbound firewall rules on each box, per §4.
- Optional, and recommended for parity: the `ZbOtelCollector` loopback collector (`infra/otel-collector/wonder/`).
- Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN
that the config will use.
### Phase 3: ConfigDb
1. Generate a **full, non-idempotent** script from the empty DB to the last migration at the candidate SHA:
`dotnet ef migrations script 0 <last> --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase`. QA learned
that a full-history `--idempotent` script fails at batch compile.
2. Review it, then run it with `sqlcmd -I -b` (QUOTED_IDENTIFIER, or the filtered indexes fail with Msg 1934)
from a box that has `sqlcmd`, using the service login. Alternatively, a DBA runs it.
3. Gate: 65 rows in `__EFMigrationsHistory`, the `zbauth` schema present (inbound API keys, Auth
`ApiKeys.SqlServer`), and the Secrets SqlServer store tables present.
4. First backup: `COPY_ONLY, CHECKSUM, COMPRESSION` + `VERIFYONLY`.
### Phase 4: MxAccessGateway ×4
**Order:** install on `app-vp03/04` first, since MXAccess is already there. Install on `PAR-VP03/04` only after the
WinPlatform deploy puts MXAccess on them (Q3 gate). Until the PAR gateways exist, the site's MxGateway connection can
point at the app-box gateways. That needs `5120` open from `10.201.75.x` to `10.220.153.x`. Afterwards, re-point
the connection to the local PAR gateways as a normal connection edit and instance redeploy.
Use `Install-MxGatewayService.ps1`, always with `-DryRun` first, and review the masked env before the real run.
- Environment `Production`, which **requires login on**. LDAPS to NAM (nam1/nam2 + `FallbackServers`), the A1P
bind, and GroupToRole for `_PROD` groups only: `APP_ADMIN_PROD`→Administrator, `APP_IDE_PROD`→Viewer,
`APP_IDE_RO_PROD`→Viewer (`ldap_mapping.md` §4.1).
- **Fresh prod secrets on each box:** `ZB_SECRETS_MASTER_KEY`, `MxGateway__ApiKeyPepper`. Nothing is copied from
QA. Choose one of two patterns and record the choice:
(a) **Recommended:** the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so
one client token works on both site gateways;
(b) independent stores, with a key per box.
- Galaxy SQL and the alarm subscription per Q2. `SqlitePath` values are absolute and outside the content root.
- Mint client keys: `scadabridge-site-prod` on the PAR gateways, plus whatever Q4 decides for the app boxes.
Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else.
- Gates: dashboard login with a NAM `_PROD` user succeeds; a user with zero roles is refused; with the client
token, `galaxy-test-connection` returns `ok:true`, and a session open/close works on each box; a bogus key
gets a typed auth error; a worker session reports ready.
### Phase 5: ScadaBridge central pair (vp03, vp04)
- Config, per node:
- `Node:Role=Central`, `NodeHostname` = the FQDN;
- seed nodes **self first** (StartupValidator enforces it, with the same spelling as `NodeHostname`);
- the ConfigDb connection string; `Secrets:Store:Provider=SqlServer` (one store shared by both centrals);
- a **shared `JwtSigningKey`** (≥ 32 random characters), shared ASP.NET Data Protection keys, and an
`ApiKeyPepper`;
- NAM LDAPS with `AllowInsecure=false` and the A1P bind;
- GroupToRole for `_PROD` only, including `APP_DEPLOY_PROD`→Deployer (`Administrator` does not include
Deployer);
- downing `leader-auto-down` (the shipped default); OTLP to loopback if Phase 2 installed the collector.
Start from the QA Central env **key names** (names only, never values) plus QA's appsettings shape. Write a
per-node diff table before anything starts.
- Recovery, as on QA: restart 5 s / 5 s / 10 s, reset 86 400, `failureflag 1` (exit 75 recovers).
- Bundle signing: generate the prod signing key (`docs/operations/bundle-signing-runbook.md`), add the QA
exporter's fingerprint to the trust list, and turn `RequireSignedBundles` **on** (this is the production
boundary the runbook describes).
- Start order: vp03, wait for ready 200, then vp04.
- Gates: ready 200 on both; active 200 on one node and 503 on the other; `required-singletons` healthy;
`bundle-signing-policy` healthy; a NAM login works in the UI and in the CLI (`--url`, A3P or a named
deployer); a zero-role user is refused at `/auth/login`, `/auth/token` and `/management`.
### Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04)
- On the central: create the `Site` entity (`SiteId` per Q8), with `GrpcNodeAAddress`/`GrpcNodeBAddress` set to
`http://<PAR-VP0x FQDN>:8083`. Write `SB-GRPC-PSK-<siteId>` through `/admin/secrets`.
- Site config:
- `Role=Site`, `SiteId`, seed nodes self first;
- `Communication:CentralGrpcEndpoints` = both centrals' `:8083`; `GrpcPsk` = `${secret:SB-GRPC-PSK-<siteId>}`;
- `LocalDb:Replication:ApiKey` byte-identical on both nodes; `PeerAddress` on **one** node only;
- absolute paths for `AuditLog:SiteWriter:DatabasePath` and the LocalDb path (QA once found the site audit DB
in System32).
- **Start the pair together**, because a site pair cannot be rolled one node at a time.
- Gates: ready 200 ×2; one node active; site online on the central health dashboard; LocalDb
`replicationConnected=true` with backlog 0; a central→site command round-trip; the site's MxGateway connection
to PAR-VP03 is connected.
### Phase 7: content and first machines
1. On QA, export a **signed** bundle containing only the selected templates, connections, inbound methods and
db-connections (Q8).
2. Import it on prod. Check that no key and no QA hostname came along: re-point each MxGateway connection to
`PAR-VP03:5120` primary and `PAR-VP04` backup, the BTDB db-connection to prod, and any OPC UA connection.
3. Mint inbound API keys on prod for MES and Delmia (`inbound-api-key-reissue.md`). Hand them over out of band.
4. Undeploy the matching machines from QA, or confirm Q9 another way. Then deploy **one machine first**, during
an idle window: verify attributes Good, alarms present, and one read-only inbound call (a `{}` body returns
400 `VALIDATION_FAILED`, which proves the key works). Then deploy the rest in batches.
5. Point MES and Delmia at prod only after step 4 passes. **This is the point of no return for consumers.**
Before it, rollback = stop the prod services.
### Phase 8: drills and handover
- Failover drills on idle machines only, copying the QA #116 procedure: on each pair, a kill drill and a
stop-and-hold drill. Expected takeover is about 8 s for a kill and about 27 s for stop-and-hold.
- Records:
- `ScadaBridge/docs/deployment/deployment-records/<date>-wonder-app-vp03-vp04-par-vp03-vp04-<sha>.md`;
- mxgw `docs/deploy/<date>-prod-<sha>.md`.
- Docs to update: `deployment.md` (§1 inventory, §2.1 hosts, a new §7.2 PROD port map, §9), the CLAUDE.md index
(on `main`, pushed), and the memory notes `wonder-environment-tiers` and `nam-ldap-cutover-handoff`.
- Overview: decide whether prod gets its own Overview, or a `Prod` registry on an existing box. It is not in
scope unless the owner asks.
## 4. Port and firewall matrix (to request in Phase 0)
| From | To | Port | Purpose |
|---|---|---|---|
| vp03 ↔ vp04 | each other | 8081 TCP | Central Akka remoting |
| PAR-VP03 ↔ PAR-VP04 | each other | 8082 TCP | Site Akka remoting |
| PAR-VP03 ↔ PAR-VP04 | each other | 8083 TCP (h2c) | LocalDb sync stream (rides the site gRPC listener) |
| PAR-VP0x | vp03, vp04 | 8083 TCP (h2c) | Site → `CentralControlService` |
| vp03, vp04 | PAR-VP0x | 8083 TCP (h2c) | Central → `SiteCommandService`, streaming, audit pull |
| PAR-VP0x (Site) | PAR-VP03/04 | 5120 TCP | ScadaBridge MxGateway adapter |
| vp03, vp04 | prod SQL | 1433 TCP | ConfigDb, Secrets, zbauth |
| all four | GR node SQL | 1433 TCP | mxgw Galaxy browse (Q2) |
| vp03, vp04, all mxgw | nam1/nam2 | 636 TCP | NAM LDAPS |
| users, MES, Delmia | vp03, vp04 | 8085 (or 443 with TLS) | Central UI, Management, Inbound API |
| admins | all four | 5130 | mxgw dashboard |
| central and site | SMTP or EWS | 587 / 443 | Notifications, if in scope |
| Mac (deploy) | all four | 5985, 445 | WinRM deploy scope + SMB (only if Q6 allows) |
The LocalDb sync row is verified: the rig's `site-a-node-a` uses `PeerAddress: http://scadabridge-site-a-b:8083`,
the existing site gRPC listener.
## 5. Rollback
- **Before Phase 7 step 5**, prod is greenfield. Rollback = stop and disable the services. QA keeps serving
every consumer.
- **After consumers are cut over**, rollback = point MES and Delmia back at QA, and redeploy the moved machines on
QA. The `.bak` folders and DB backup follow the same pattern as QA (binaries + `COPY_ONLY` backup per window).
- Every later prod upgrade uses the QA pack pattern: stop both sites, stop both centrals, migrate, swap, start
the centrals, start the sites.
## 6. Risks
| Risk | Mitigation |
|---|---|
| The central↔site boundary (`8083`) is blocked or filtered in one direction only. QA never exercised it, because it was co-located | Phase 0 firewall ticket; Phase 6 round-trip gate in **both** directions |
| One machine driven by two stacks (Q9) | Undeploy from QA before the prod deploy |
| `_PROD` groups grant wider access than intended | T6 membership review before mapping |
| A service-account lockout breaks every app | A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule) |
| The mxgw upgrade script aborts on a slow start | Phase 1 step 1 fix |
| The security software flags remote exec | Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert |