Files
scadaproj/docs/plans/2026-11-19-prod-scadabridge-mxgw-install.md
T
Joseph Doherty 91d35edb9c docs(plan): PROD install of ScadaBridge + mxaccessgw (app-vp03/04 central, par-vp03/04 site)
Plan only, nothing executed. Records owner decisions (mxgw on all four boxes,
current main via QA, NAM AD LDAPS day one, signed QA bundle), the SMB-verified
box facts (admin account, subnets, .NET, MXAccess 6000.0045.2657.1; PAR boxes
wait on the WinPlatform deploy), phases 0-8, port matrix, rollback and open
questions.
2026-11-19 14:44:26 -05:00

20 KiB
Raw Blame History

PROD install: ScadaBridge + MxAccessGateway (plan, 2026-11-19)

State: PLAN ONLY. Nothing has been executed, and no prod box has been touched. This is the first production tier for the family. Until now deployment.md said "no production box has been identified".

The DEPLOY HOLD still applies to QA and PROD (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every phase below that touches QA or a prod box needs its own explicit owner go.

1. Target topology

Host Role ScadaBridge MxAccessGateway
wonder-app-vp03 Central A ScadaBridge-Central (UI/Management/Inbound API :8085, Akka :8081, central gRPC :8083) MxAccessGw :5120 gRPC / :5130 dashboard
wonder-app-vp04 Central B ScadaBridge-Central (same ports) MxAccessGw (same ports)
WONDER-PAR-VP03 Site node A ScadaBridge-Site (Akka :8082, gRPC :8083, health :8084) MxAccessGw (same ports)
WONDER-PAR-VP04 Site node B ScadaBridge-Site (same ports) MxAccessGw (same ports)

This differs from QA. On QA, vq03/vq04 each run a central node and a site node. In prod the central pair and the site pair are on different boxes, and possibly on different network segments (app vs PAR). The central↔site gRPC control plane (:8083, both directions, PSK-authenticated) therefore crosses a network boundary for the first time. It is the main new risk in this install.

Decisions recorded (owner, 2026-11-19):

  • D-P1 mxgw on all four boxes. Each box needs AVEVA System Platform with MXAccess registered (LMXProxy.LMXProxyServer). The site pair's MxGateway connections use the PAR gateways. The purpose of the central-box gateways has to be confirmed (see Q4).
  • D-P2 builds: current main, promoted through QA first. ScadaBridge 2e0ce81e (includes #144/#145, which refuse zero-role logins) and mxgw e74918cc. Prod gets the same zips, byte-identical to the ones that soaked on QA.
  • D-P3 NAM AD over LDAPS from day one. This uses _PROD groups only and separate PROD accounts. There is no GLAuth in prod.
  • D-P4 content: a signed config bundle from QA, limited to the machines selected for prod. Prod connections are re-pointed at the prod gateways, and prod mints its own inbound API keys.

2. Open questions (each blocks the phase noted)

# Question Blocks
Q1 Which SQL Server hosts the prod ScadaBridge ConfigDb? Who creates the DB and the scadabridge_svc login? Phase 3
Q2 Where is the prod Galaxy Repository (ZB DB) and its GR node, and which alarm area do the gateways subscribe to (\\<GR node>\Galaxy!<area>)? Phase 4
Q3 Partly answered over SMB (2026-11-19, read-only, table below). ⚠ The PAR boxes have no MXAccess. app-vp03/04 have a full Application Server install: ArchestrA\Framework\Bin holds 562 entries, including LmxProxy.dll. PAR-VP03/04 hold only 49 entries there, with no LmxProxy.dll; they carry OI Server, Historian Client and the common components. Reference: prod runtime node wonder-bd5-vp03 has MXAccess with aaEngine.exe and LmxProxy.dll 6000.0045.2657.1 (131 entries), byte-identical to vp03 and to QA vq03; that is the level the PAR boxes need. Owner, 2026-11-19: WinPlatform is not deployed to the PAR boxes yet. Deploying it from the prod Galaxy installs MXAccess there. So mxgw on PAR is gated on that deploy (an AVEVA/Galaxy step, outside this plan). Gate check over SMB: C$\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll is present at 6000.0045.2657.1 (SHA prefix bd75a2fde222). Until then, mxgw cannot run on the PAR boxes: it needs a WinPlatform deployed to the box. The alternative is to point the site's MxGateway connections at the app-box gateways (see Q4). Still open: NSSM (none found under Program Files) and free disk. Phase 2, Phase 4
Q4 What consumes the gateways on app-vp03/04? ScadaBridge centrals don't talk to mxgw. If the answer is "nothing yet" (for example, a future OtOpcUa), install them but mint no client keys. Phase 4
Q5 Network: DNS resolved from the Mac on 2026-11-19 (table below). The centrals and the sites are on different subnets (10.220.153.x vs 10.201.75.x), so a firewall between them is likely. Find out whether one exists. See the port matrix in §4. Phase 0
Q6 Remote deploy from the Mac: the account is settled (below). Still open: whether WinRM 5985 and SMB 445 are open from the Mac to the four boxes, and whether a prod change needs a corporate change ticket. The security software tripped on WinRM before (2026-09-11). Every phase on a box

Hosts (DNS from the Mac, 2026-11-19):

FQDN IP
wonder-app-vp03.zmr.zimmer.com 10.220.153.205
wonder-app-vp04.zmr.zimmer.com 10.220.153.210
wonder-par-vp03.zmr.zimmer.com 10.201.75.86
wonder-par-vp04.zmr.zimmer.com 10.201.75.87

QA's vq03/vq04 sit on 10.220.158.x.

Remote-access account: QA and PROD boxes outside the wonder DEV/QA set need the admin account NAM\adm_dohertj2. NAM\dohertj2 gets STATUS_ACCESS_DENIED on the C$ share of the DNC QA/PROD boxes. That account is already in the winrm Keychain as profiles dnc-vq01 (dnc-app-vq01) and dnc-vp01 (dnc-app-vp01), both NTLM over HTTP 5985. The prod profiles are vp03, vp04, parvp03 and parvp04, as NAM\adm_dohertj2, with the password copied Keychain→Keychain from winrm-cli/dnc-vp01 (the same method used for sqlvd03). Whether adm_dohertj2 is a local admin on the wonder prod boxes is unverified.

SMB check (2026-11-19, winrm smb ls only, no remote exec):

  • NAM\adm_dohertj2 opens C$ and E$ on all four boxes, so it has admin-share rights. None of them has a D$ share.
  • No E:\ApiInstall exists yet on any box, so this is greenfield. Each box has E:\2023R2Install (AVEVA 2023 R2 media) and E:\Hotfixes. vp03's E: also holds ProdPromotions, ServiceIntegrations and Backups.
  • .NET: only 6.0.25 and 8.0.0, so the builds must be self-contained.
  • Program Files\Microsoft SQL Server is present on vp03 and parvp03. Whether those instances are what Q1/Q2 need is unverified.

DELMIA prod is dnc-app-vp01 (10.201.207.106). For Q8, that box is the prod Delmia caller of ScadaBridge's DelmiaRecipeDownload. Prod runs an older Downloader than QA (delmia project memory), so check the prod integration's shape before you assume it matches QA's. | Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs zimmer-rootca in the machine store (cutover-plan D4), and AllowInsecure=false in prod. | Phases 4–5 | | Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the SiteId? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 | | Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 |

3. Phases

Phase 0: approvals and external prerequisites (owner + IT, before any box work)

  1. Ticket T6 (nam_ad_requests.md §6), filed with IT:
    • A1P svc_zbmomww_prod: the LDAP lookup account, no lockout, read-only;
    • A3P svc_zbmomww_auto_prod, if the owner allows PROD automation (plan D3);
    • G3 UG_WW_APP_DEPLOY_PROD;
    • a membership review of UG_WW_APP_ADMIN_PROD, APP_IDE_PROD, APP_IDE_RO_PROD and BTMGR_RO_PROD. These groups were built for System Platform, so mapping them grants their members app roles in prod.
  2. Extend scadaproj#58 for A1P (comment). File a cross-repo umbrella issue in scadaproj, plus a type:deploy-gate issue in ScadaBridge and in mxaccessgw, cross-linked.
  3. Firewall requests, from the §4 matrix.
  4. SQL (Q1): an empty ScadaBridge DB, the scadabridge_svc login as db_owner, and a backup target.
  5. Certificates (Q7).
  6. Access (Q6). If the Mac path is allowed, add winrm profiles vp03, vp04, parvp03 and parvp04 with Keychain credentials, then run an SMB-only reachability check (winrm smb ls). No remote exec before the owner's go.
  7. Owner go for QA promotion (Phase 1).

Phase 1: candidate builds proven on QA

  1. Fix mxgw scripts/deploy/Upgrade-MxGatewayService.ps1 first. nssm start returning SERVICE_START_PENDING makes it abort and auto-roll-back. The fix is the try/catch and the Wait-State already used on vd03 (memory note). Merge it to mxgw main. If it lands after e74918cc, the candidate SHA moves, and the build is re-proven on DEV first.
  2. Build once, then reuse the outputs for QA and PROD:
    • ScadaBridge Host and CLI: dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false;
    • mxgw Server: self-contained win-x64;
    • mxgw x86 Worker on windev, at the same SHA (-p:Platform=x86 -p:SourceRevisionId=<sha>). The upgrade script refuses mixed SHAs. Record a SHA-256 for every zip.
  3. Deploy to QA with the existing 061432c8-style pack. ScadaBridge needs no migration (061432c8..2e0ce81e has no migration change; the schema stays at 65). mxgw rolls one box at a time.
  4. Soak on QA for at least 3 working days. The gates are the QA record's set: ready 200 ×4, one active per pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative control).
  5. Optionally, do QA's NAM cutover (plan Phases A–7) before prod. Recommended. QA is then the first environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first.

Phase 2: box prep (all four, no service started)

  • Layout as on QA: E:\ApiInstall\{ScadaBridge,MxAccessGateway}\{app,cli|worker,data,logs}, with staging in E:\psrem-stage\<app>-<sha>\.
  • NSSM for mxgw. ScadaBridge services are created with sc.exe (installation guide), env in the service key Environment (REG_MULTI_SZ).
  • KEK folders C:\ProgramData\ZB\kek\ set to Administrators:F + SYSTEM:F, with inheritance off.
  • Inbound firewall rules on each box, per §4.
  • Optional, and recommended for parity: the ZbOtelCollector loopback collector (infra/otel-collector/wonder/).
  • Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN that the config will use.

Phase 3: ConfigDb

  1. Generate a full, non-idempotent script from the empty DB to the last migration at the candidate SHA: dotnet ef migrations script 0 <last> --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase. QA learned that a full-history --idempotent script fails at batch compile.
  2. Review it, then run it with sqlcmd -I -b (QUOTED_IDENTIFIER, or the filtered indexes fail with Msg 1934) from a box that has sqlcmd, using the service login. Alternatively, a DBA runs it.
  3. Gate: 65 rows in __EFMigrationsHistory, the zbauth schema present (inbound API keys, Auth ApiKeys.SqlServer), and the Secrets SqlServer store tables present.
  4. First backup: COPY_ONLY, CHECKSUM, COMPRESSION + VERIFYONLY.

Phase 4: MxAccessGateway ×4

Order: install on app-vp03/04 first, since MXAccess is already there. Install on PAR-VP03/04 only after the WinPlatform deploy puts MXAccess on them (Q3 gate). Until the PAR gateways exist, the site's MxGateway connection can point at the app-box gateways. That needs 5120 open from 10.201.75.x to 10.220.153.x. Afterwards, re-point the connection to the local PAR gateways as a normal connection edit and instance redeploy.

Use Install-MxGatewayService.ps1, always with -DryRun first, and review the masked env before the real run.

  • Environment Production, which requires login on. LDAPS to NAM (nam1/nam2 + FallbackServers), the A1P bind, and GroupToRole for _PROD groups only: APP_ADMIN_PROD→Administrator, APP_IDE_PROD→Viewer, APP_IDE_RO_PROD→Viewer (ldap_mapping.md §4.1).
  • Fresh prod secrets on each box: ZB_SECRETS_MASTER_KEY, MxGateway__ApiKeyPepper. Nothing is copied from QA. Choose one of two patterns and record the choice: (a) Recommended: the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so one client token works on both site gateways; (b) independent stores, with a key per box.
  • Galaxy SQL and the alarm subscription per Q2. SqlitePath values are absolute and outside the content root.
  • Mint client keys: scadabridge-site-prod on the PAR gateways, plus whatever Q4 decides for the app boxes. Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else.
  • Gates: dashboard login with a NAM _PROD user succeeds; a user with zero roles is refused; with the client token, galaxy-test-connection returns ok:true, and a session open/close works on each box; a bogus key gets a typed auth error; a worker session reports ready.

Phase 5: ScadaBridge central pair (vp03, vp04)

  • Config, per node:
    • Node:Role=Central, NodeHostname = the FQDN;
    • seed nodes self first (StartupValidator enforces it, with the same spelling as NodeHostname);
    • the ConfigDb connection string; Secrets:Store:Provider=SqlServer (one store shared by both centrals);
    • a shared JwtSigningKey (≥ 32 random characters), shared ASP.NET Data Protection keys, and an ApiKeyPepper;
    • NAM LDAPS with AllowInsecure=false and the A1P bind;
    • GroupToRole for _PROD only, including APP_DEPLOY_PROD→Deployer (Administrator does not include Deployer);
    • downing leader-auto-down (the shipped default); OTLP to loopback if Phase 2 installed the collector. Start from the QA Central env key names (names only, never values) plus QA's appsettings shape. Write a per-node diff table before anything starts.
  • Recovery, as on QA: restart 5 s / 5 s / 10 s, reset 86 400, failureflag 1 (exit 75 recovers).
  • Bundle signing: generate the prod signing key (docs/operations/bundle-signing-runbook.md), add the QA exporter's fingerprint to the trust list, and turn RequireSignedBundles on (this is the production boundary the runbook describes).
  • Start order: vp03, wait for ready 200, then vp04.
  • Gates: ready 200 on both; active 200 on one node and 503 on the other; required-singletons healthy; bundle-signing-policy healthy; a NAM login works in the UI and in the CLI (--url, A3P or a named deployer); a zero-role user is refused at /auth/login, /auth/token and /management.

Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04)

  • On the central: create the Site entity (SiteId per Q8), with GrpcNodeAAddress/GrpcNodeBAddress set to http://<PAR-VP0x FQDN>:8083. Write SB-GRPC-PSK-<siteId> through /admin/secrets.
  • Site config:
    • Role=Site, SiteId, seed nodes self first;
    • Communication:CentralGrpcEndpoints = both centrals' :8083; GrpcPsk = ${secret:SB-GRPC-PSK-<siteId>};
    • LocalDb:Replication:ApiKey byte-identical on both nodes; PeerAddress on one node only;
    • absolute paths for AuditLog:SiteWriter:DatabasePath and the LocalDb path (QA once found the site audit DB in System32).
  • Start the pair together, because a site pair cannot be rolled one node at a time.
  • Gates: ready 200 ×2; one node active; site online on the central health dashboard; LocalDb replicationConnected=true with backlog 0; a central→site command round-trip; the site's MxGateway connection to PAR-VP03 is connected.

Phase 7: content and first machines

  1. On QA, export a signed bundle containing only the selected templates, connections, inbound methods and db-connections (Q8).
  2. Import it on prod. Check that no key and no QA hostname came along: re-point each MxGateway connection to PAR-VP03:5120 primary and PAR-VP04 backup, the BTDB db-connection to prod, and any OPC UA connection.
  3. Mint inbound API keys on prod for MES and Delmia (inbound-api-key-reissue.md). Hand them over out of band.
  4. Undeploy the matching machines from QA, or confirm Q9 another way. Then deploy one machine first, during an idle window: verify attributes Good, alarms present, and one read-only inbound call (a {} body returns 400 VALIDATION_FAILED, which proves the key works). Then deploy the rest in batches.
  5. Point MES and Delmia at prod only after step 4 passes. This is the point of no return for consumers. Before it, rollback = stop the prod services.

Phase 8: drills and handover

  • Failover drills on idle machines only, copying the QA #116 procedure: on each pair, a kill drill and a stop-and-hold drill. Expected takeover is about 8 s for a kill and about 27 s for stop-and-hold.
  • Records:
    • ScadaBridge/docs/deployment/deployment-records/<date>-wonder-app-vp03-vp04-par-vp03-vp04-<sha>.md;
    • mxgw docs/deploy/<date>-prod-<sha>.md.
  • Docs to update: deployment.md (§1 inventory, §2.1 hosts, a new §7.2 PROD port map, §9), the CLAUDE.md index (on main, pushed), and the memory notes wonder-environment-tiers and nam-ldap-cutover-handoff.
  • Overview: decide whether prod gets its own Overview, or a Prod registry on an existing box. It is not in scope unless the owner asks.

4. Port and firewall matrix (to request in Phase 0)

From To Port Purpose
vp03 ↔ vp04 each other 8081 TCP Central Akka remoting
PAR-VP03 ↔ PAR-VP04 each other 8082 TCP Site Akka remoting
PAR-VP03 ↔ PAR-VP04 each other 8083 TCP (h2c) LocalDb sync stream (rides the site gRPC listener)
PAR-VP0x vp03, vp04 8083 TCP (h2c) Site → CentralControlService
vp03, vp04 PAR-VP0x 8083 TCP (h2c) Central → SiteCommandService, streaming, audit pull
PAR-VP0x (Site) PAR-VP03/04 5120 TCP ScadaBridge MxGateway adapter
vp03, vp04 prod SQL 1433 TCP ConfigDb, Secrets, zbauth
all four GR node SQL 1433 TCP mxgw Galaxy browse (Q2)
vp03, vp04, all mxgw nam1/nam2 636 TCP NAM LDAPS
users, MES, Delmia vp03, vp04 8085 (or 443 with TLS) Central UI, Management, Inbound API
admins all four 5130 mxgw dashboard
central and site SMTP or EWS 587 / 443 Notifications, if in scope
Mac (deploy) all four 5985, 445 WinRM deploy scope + SMB (only if Q6 allows)

The LocalDb sync row is verified: the rig's site-a-node-a uses PeerAddress: http://scadabridge-site-a-b:8083, the existing site gRPC listener.

5. Rollback

  • Before Phase 7 step 5, prod is greenfield. Rollback = stop and disable the services. QA keeps serving every consumer.
  • After consumers are cut over, rollback = point MES and Delmia back at QA, and redeploy the moved machines on QA. The .bak folders and DB backup follow the same pattern as QA (binaries + COPY_ONLY backup per window).
  • Every later prod upgrade uses the QA pack pattern: stop both sites, stop both centrals, migrate, swap, start the centrals, start the sites.

6. Risks

Risk Mitigation
The central↔site boundary (8083) is blocked or filtered in one direction only. QA never exercised it, because it was co-located Phase 0 firewall ticket; Phase 6 round-trip gate in both directions
One machine driven by two stacks (Q9) Undeploy from QA before the prod deploy
_PROD groups grant wider access than intended T6 membership review before mapping
A service-account lockout breaks every app A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule)
The mxgw upgrade script aborts on a slow start Phase 1 step 1 fix
The security software flags remote exec Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert