Plan only, nothing executed. Records owner decisions (mxgw on all four boxes, current main via QA, NAM AD LDAPS day one, signed QA bundle), the SMB-verified box facts (admin account, subnets, .NET, MXAccess 6000.0045.2657.1; PAR boxes wait on the WinPlatform deploy), phases 0-8, port matrix, rollback and open questions.
20 KiB
PROD install: ScadaBridge + MxAccessGateway (plan, 2026-11-19)
State: PLAN ONLY. Nothing has been executed, and no prod box has been touched. This is the first production
tier for the family. Until now deployment.md said "no production box has been identified".
The DEPLOY HOLD still applies to QA and PROD (owner, 2026-11-17; lifted for DEV only on 2026-11-19). Every phase below that touches QA or a prod box needs its own explicit owner go.
1. Target topology
| Host | Role | ScadaBridge | MxAccessGateway |
|---|---|---|---|
wonder-app-vp03 |
Central A | ScadaBridge-Central (UI/Management/Inbound API :8085, Akka :8081, central gRPC :8083) |
MxAccessGw :5120 gRPC / :5130 dashboard |
wonder-app-vp04 |
Central B | ScadaBridge-Central (same ports) |
MxAccessGw (same ports) |
WONDER-PAR-VP03 |
Site node A | ScadaBridge-Site (Akka :8082, gRPC :8083, health :8084) |
MxAccessGw (same ports) |
WONDER-PAR-VP04 |
Site node B | ScadaBridge-Site (same ports) |
MxAccessGw (same ports) |
This differs from QA. On QA, vq03/vq04 each run a central node and a site node. In prod the central pair
and the site pair are on different boxes, and possibly on different network segments (app vs PAR). The
central↔site gRPC control plane (:8083, both directions, PSK-authenticated) therefore crosses a network
boundary for the first time. It is the main new risk in this install.
Decisions recorded (owner, 2026-11-19):
- D-P1 mxgw on all four boxes. Each box needs AVEVA System Platform with MXAccess registered
(
LMXProxy.LMXProxyServer). The site pair's MxGateway connections use the PAR gateways. The purpose of the central-box gateways has to be confirmed (see Q4). - D-P2 builds: current
main, promoted through QA first. ScadaBridge2e0ce81e(includes #144/#145, which refuse zero-role logins) and mxgwe74918cc. Prod gets the same zips, byte-identical to the ones that soaked on QA. - D-P3 NAM AD over LDAPS from day one. This uses
_PRODgroups only and separate PROD accounts. There is no GLAuth in prod. - D-P4 content: a signed config bundle from QA, limited to the machines selected for prod. Prod connections are re-pointed at the prod gateways, and prod mints its own inbound API keys.
2. Open questions (each blocks the phase noted)
| # | Question | Blocks |
|---|---|---|
| Q1 | Which SQL Server hosts the prod ScadaBridge ConfigDb? Who creates the DB and the scadabridge_svc login? |
Phase 3 |
| Q2 | Where is the prod Galaxy Repository (ZB DB) and its GR node, and which alarm area do the gateways subscribe to (\\<GR node>\Galaxy!<area>)? |
Phase 4 |
| Q3 | Partly answered over SMB (2026-11-19, read-only, table below). ⚠ The PAR boxes have no MXAccess. app-vp03/04 have a full Application Server install: ArchestrA\Framework\Bin holds 562 entries, including LmxProxy.dll. PAR-VP03/04 hold only 49 entries there, with no LmxProxy.dll; they carry OI Server, Historian Client and the common components. Reference: prod runtime node wonder-bd5-vp03 has MXAccess with aaEngine.exe and LmxProxy.dll 6000.0045.2657.1 (131 entries), byte-identical to vp03 and to QA vq03; that is the level the PAR boxes need. Owner, 2026-11-19: WinPlatform is not deployed to the PAR boxes yet. Deploying it from the prod Galaxy installs MXAccess there. So mxgw on PAR is gated on that deploy (an AVEVA/Galaxy step, outside this plan). Gate check over SMB: C$\Program Files (x86)\ArchestrA\Framework\Bin\LmxProxy.dll is present at 6000.0045.2657.1 (SHA prefix bd75a2fde222). Until then, mxgw cannot run on the PAR boxes: it needs a WinPlatform deployed to the box. The alternative is to point the site's MxGateway connections at the app-box gateways (see Q4). Still open: NSSM (none found under Program Files) and free disk. |
Phase 2, Phase 4 |
| Q4 | What consumes the gateways on app-vp03/04? ScadaBridge centrals don't talk to mxgw. If the answer is "nothing yet" (for example, a future OtOpcUa), install them but mint no client keys. |
Phase 4 |
| Q5 | Network: DNS resolved from the Mac on 2026-11-19 (table below). The centrals and the sites are on different subnets (10.220.153.x vs 10.201.75.x), so a firewall between them is likely. Find out whether one exists. See the port matrix in §4. |
Phase 0 |
| Q6 | Remote deploy from the Mac: the account is settled (below). Still open: whether WinRM 5985 and SMB 445 are open from the Mac to the four boxes, and whether a prod change needs a corporate change ticket. The security software tripped on WinRM before (2026-09-11). | Every phase on a box |
Hosts (DNS from the Mac, 2026-11-19):
| FQDN | IP |
|---|---|
wonder-app-vp03.zmr.zimmer.com |
10.220.153.205 |
wonder-app-vp04.zmr.zimmer.com |
10.220.153.210 |
wonder-par-vp03.zmr.zimmer.com |
10.201.75.86 |
wonder-par-vp04.zmr.zimmer.com |
10.201.75.87 |
QA's vq03/vq04 sit on 10.220.158.x.
Remote-access account: QA and PROD boxes outside the wonder DEV/QA set need the admin account
NAM\adm_dohertj2. NAM\dohertj2 gets STATUS_ACCESS_DENIED on the C$ share of the DNC QA/PROD
boxes. That account is already in the winrm Keychain as profiles dnc-vq01 (dnc-app-vq01) and dnc-vp01
(dnc-app-vp01), both NTLM over HTTP 5985. The prod profiles are vp03, vp04, parvp03 and parvp04, as
NAM\adm_dohertj2, with the password copied Keychain→Keychain from winrm-cli/dnc-vp01 (the same method used
for sqlvd03). Whether adm_dohertj2 is a local admin on the wonder prod boxes is unverified.
SMB check (2026-11-19, winrm smb ls only, no remote exec):
NAM\adm_dohertj2opensC$andE$on all four boxes, so it has admin-share rights. None of them has aD$share.- No
E:\ApiInstallexists yet on any box, so this is greenfield. Each box hasE:\2023R2Install(AVEVA 2023 R2 media) andE:\Hotfixes.vp03'sE:also holdsProdPromotions,ServiceIntegrationsandBackups. - .NET: only 6.0.25 and 8.0.0, so the builds must be self-contained.
Program Files\Microsoft SQL Serveris present on vp03 and parvp03. Whether those instances are what Q1/Q2 need is unverified.
DELMIA prod is dnc-app-vp01 (10.201.207.106). For Q8, that box is the prod Delmia caller of
ScadaBridge's DelmiaRecipeDownload. Prod runs an older Downloader than QA (delmia project memory), so check
the prod integration's shape before you assume it matches QA's.
| Q7 | TLS: is there a corporate cert for the Central UI (HTTPS) and for the mxgw dashboard? LDAPS needs zimmer-rootca in the machine store (cutover-plan D4), and AllowInsecure=false in prod. | Phases 4–5 |
| Q8 | Which machines (SAPIDs / Galaxy objects) go to prod on day one? What is the SiteId? Do the BTDB connection, MES and Delmia endpoints, and SMTP/EWS have prod equivalents? | Phase 7 |
| Q9 | Will any machine be driven by two ScadaBridge stacks at once (QA and prod)? Prod writes to the real Galaxy. A machine moved to prod should be undeployed from QA, or QA must be on a different Galaxy. | Phase 7 |
3. Phases
Phase 0: approvals and external prerequisites (owner + IT, before any box work)
- Ticket T6 (
nam_ad_requests.md§6), filed with IT:- A1P
svc_zbmomww_prod: the LDAP lookup account, no lockout, read-only; - A3P
svc_zbmomww_auto_prod, if the owner allows PROD automation (plan D3); - G3
UG_WW_APP_DEPLOY_PROD; - a membership review of
UG_WW_APP_ADMIN_PROD,APP_IDE_PROD,APP_IDE_RO_PRODandBTMGR_RO_PROD. These groups were built for System Platform, so mapping them grants their members app roles in prod.
- A1P
- Extend scadaproj#58 for A1P (comment). File a
cross-repoumbrella issue in scadaproj, plus atype:deploy-gateissue in ScadaBridge and in mxaccessgw, cross-linked. - Firewall requests, from the §4 matrix.
- SQL (Q1): an empty
ScadaBridgeDB, thescadabridge_svclogin asdb_owner, and a backup target. - Certificates (Q7).
- Access (Q6). If the Mac path is allowed, add
winrmprofilesvp03,vp04,parvp03andparvp04with Keychain credentials, then run an SMB-only reachability check (winrm smb ls). No remote exec before the owner's go. - Owner go for QA promotion (Phase 1).
Phase 1: candidate builds proven on QA
- Fix mxgw
scripts/deploy/Upgrade-MxGatewayService.ps1first.nssm startreturningSERVICE_START_PENDINGmakes it abort and auto-roll-back. The fix is thetry/catchand theWait-Statealready used on vd03 (memory note). Merge it to mxgwmain. If it lands aftere74918cc, the candidate SHA moves, and the build is re-proven on DEV first. - Build once, then reuse the outputs for QA and PROD:
- ScadaBridge Host and CLI:
dotnet publish -c Release -r win-x64 --self-contained true -p:PublishSingleFile=false; - mxgw Server: self-contained win-x64;
- mxgw x86 Worker on windev, at the same SHA (
-p:Platform=x86 -p:SourceRevisionId=<sha>). The upgrade script refuses mixed SHAs. Record a SHA-256 for every zip.
- ScadaBridge Host and CLI:
- Deploy to QA with the existing 061432c8-style pack. ScadaBridge needs no migration (
061432c8..2e0ce81ehas no migration change; the schema stays at 65). mxgw rolls one box at a time. - Soak on QA for at least 3 working days. The gates are the QA record's set: ready 200 ×4, one active per pair, MxGateway 2196/2196 good, LocalDb sync connected, and zero-role login refused (#144/#145 negative control).
- Optionally, do QA's NAM cutover (plan Phases A–7) before prod. Recommended. QA is then the first environment that proves NAM AD over LDAPS on these exact builds, so prod is not the first.
Phase 2: box prep (all four, no service started)
- Layout as on QA:
E:\ApiInstall\{ScadaBridge,MxAccessGateway}\{app,cli|worker,data,logs}, with staging inE:\psrem-stage\<app>-<sha>\. - NSSM for mxgw. ScadaBridge services are created with
sc.exe(installation guide), env in the service keyEnvironment(REG_MULTI_SZ). - KEK folders
C:\ProgramData\ZB\kek\set toAdministrators:F + SYSTEM:F, with inheritance off. - Inbound firewall rules on each box, per §4.
- Optional, and recommended for parity: the
ZbOtelCollectorloopback collector (infra/otel-collector/wonder/). - Gate: SHA re-verified on each box, MXAccess COM registered (Q3), and the boxes resolve each other by the FQDN that the config will use.
Phase 3: ConfigDb
- Generate a full, non-idempotent script from the empty DB to the last migration at the candidate SHA:
dotnet ef migrations script 0 <last> --project src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase. QA learned that a full-history--idempotentscript fails at batch compile. - Review it, then run it with
sqlcmd -I -b(QUOTED_IDENTIFIER, or the filtered indexes fail with Msg 1934) from a box that hassqlcmd, using the service login. Alternatively, a DBA runs it. - Gate: 65 rows in
__EFMigrationsHistory, thezbauthschema present (inbound API keys, AuthApiKeys.SqlServer), and the Secrets SqlServer store tables present. - First backup:
COPY_ONLY, CHECKSUM, COMPRESSION+VERIFYONLY.
Phase 4: MxAccessGateway ×4
Order: install on app-vp03/04 first, since MXAccess is already there. Install on PAR-VP03/04 only after the
WinPlatform deploy puts MXAccess on them (Q3 gate). Until the PAR gateways exist, the site's MxGateway connection can
point at the app-box gateways. That needs 5120 open from 10.201.75.x to 10.220.153.x. Afterwards, re-point
the connection to the local PAR gateways as a normal connection edit and instance redeploy.
Use Install-MxGatewayService.ps1, always with -DryRun first, and review the masked env before the real run.
- Environment
Production, which requires login on. LDAPS to NAM (nam1/nam2 +FallbackServers), the A1P bind, and GroupToRole for_PRODgroups only:APP_ADMIN_PROD→Administrator,APP_IDE_PROD→Viewer,APP_IDE_RO_PROD→Viewer (ldap_mapping.md§4.1). - Fresh prod secrets on each box:
ZB_SECRETS_MASTER_KEY,MxGateway__ApiKeyPepper. Nothing is copied from QA. Choose one of two patterns and record the choice: (a) Recommended: the PAR pair shares one pepper and one API-key store (QA's 2026-08-14 clone pattern), so one client token works on both site gateways; (b) independent stores, with a key per box. - Galaxy SQL and the alarm subscription per Q2.
SqlitePathvalues are absolute and outside the content root. - Mint client keys:
scadabridge-site-prodon the PAR gateways, plus whatever Q4 decides for the app boxes. Tokens go to the Keychain and to ScadaBridge's secret store, and nowhere else. - Gates: dashboard login with a NAM
_PRODuser succeeds; a user with zero roles is refused; with the client token,galaxy-test-connectionreturnsok:true, and a session open/close works on each box; a bogus key gets a typed auth error; a worker session reports ready.
Phase 5: ScadaBridge central pair (vp03, vp04)
- Config, per node:
Node:Role=Central,NodeHostname= the FQDN;- seed nodes self first (StartupValidator enforces it, with the same spelling as
NodeHostname); - the ConfigDb connection string;
Secrets:Store:Provider=SqlServer(one store shared by both centrals); - a shared
JwtSigningKey(≥ 32 random characters), shared ASP.NET Data Protection keys, and anApiKeyPepper; - NAM LDAPS with
AllowInsecure=falseand the A1P bind; - GroupToRole for
_PRODonly, includingAPP_DEPLOY_PROD→Deployer (Administratordoes not include Deployer); - downing
leader-auto-down(the shipped default); OTLP to loopback if Phase 2 installed the collector. Start from the QA Central env key names (names only, never values) plus QA's appsettings shape. Write a per-node diff table before anything starts.
- Recovery, as on QA: restart 5 s / 5 s / 10 s, reset 86 400,
failureflag 1(exit 75 recovers). - Bundle signing: generate the prod signing key (
docs/operations/bundle-signing-runbook.md), add the QA exporter's fingerprint to the trust list, and turnRequireSignedBundleson (this is the production boundary the runbook describes). - Start order: vp03, wait for ready 200, then vp04.
- Gates: ready 200 on both; active 200 on one node and 503 on the other;
required-singletonshealthy;bundle-signing-policyhealthy; a NAM login works in the UI and in the CLI (--url, A3P or a named deployer); a zero-role user is refused at/auth/login,/auth/tokenand/management.
Phase 6: ScadaBridge site pair (PAR-VP03, PAR-VP04)
- On the central: create the
Siteentity (SiteIdper Q8), withGrpcNodeAAddress/GrpcNodeBAddressset tohttp://<PAR-VP0x FQDN>:8083. WriteSB-GRPC-PSK-<siteId>through/admin/secrets. - Site config:
Role=Site,SiteId, seed nodes self first;Communication:CentralGrpcEndpoints= both centrals':8083;GrpcPsk=${secret:SB-GRPC-PSK-<siteId>};LocalDb:Replication:ApiKeybyte-identical on both nodes;PeerAddresson one node only;- absolute paths for
AuditLog:SiteWriter:DatabasePathand the LocalDb path (QA once found the site audit DB in System32).
- Start the pair together, because a site pair cannot be rolled one node at a time.
- Gates: ready 200 ×2; one node active; site online on the central health dashboard; LocalDb
replicationConnected=truewith backlog 0; a central→site command round-trip; the site's MxGateway connection to PAR-VP03 is connected.
Phase 7: content and first machines
- On QA, export a signed bundle containing only the selected templates, connections, inbound methods and db-connections (Q8).
- Import it on prod. Check that no key and no QA hostname came along: re-point each MxGateway connection to
PAR-VP03:5120primary andPAR-VP04backup, the BTDB db-connection to prod, and any OPC UA connection. - Mint inbound API keys on prod for MES and Delmia (
inbound-api-key-reissue.md). Hand them over out of band. - Undeploy the matching machines from QA, or confirm Q9 another way. Then deploy one machine first, during
an idle window: verify attributes Good, alarms present, and one read-only inbound call (a
{}body returns 400VALIDATION_FAILED, which proves the key works). Then deploy the rest in batches. - Point MES and Delmia at prod only after step 4 passes. This is the point of no return for consumers. Before it, rollback = stop the prod services.
Phase 8: drills and handover
- Failover drills on idle machines only, copying the QA #116 procedure: on each pair, a kill drill and a stop-and-hold drill. Expected takeover is about 8 s for a kill and about 27 s for stop-and-hold.
- Records:
ScadaBridge/docs/deployment/deployment-records/<date>-wonder-app-vp03-vp04-par-vp03-vp04-<sha>.md;- mxgw
docs/deploy/<date>-prod-<sha>.md.
- Docs to update:
deployment.md(§1 inventory, §2.1 hosts, a new §7.2 PROD port map, §9), the CLAUDE.md index (onmain, pushed), and the memory noteswonder-environment-tiersandnam-ldap-cutover-handoff. - Overview: decide whether prod gets its own Overview, or a
Prodregistry on an existing box. It is not in scope unless the owner asks.
4. Port and firewall matrix (to request in Phase 0)
| From | To | Port | Purpose |
|---|---|---|---|
| vp03 ↔ vp04 | each other | 8081 TCP | Central Akka remoting |
| PAR-VP03 ↔ PAR-VP04 | each other | 8082 TCP | Site Akka remoting |
| PAR-VP03 ↔ PAR-VP04 | each other | 8083 TCP (h2c) | LocalDb sync stream (rides the site gRPC listener) |
| PAR-VP0x | vp03, vp04 | 8083 TCP (h2c) | Site → CentralControlService |
| vp03, vp04 | PAR-VP0x | 8083 TCP (h2c) | Central → SiteCommandService, streaming, audit pull |
| PAR-VP0x (Site) | PAR-VP03/04 | 5120 TCP | ScadaBridge MxGateway adapter |
| vp03, vp04 | prod SQL | 1433 TCP | ConfigDb, Secrets, zbauth |
| all four | GR node SQL | 1433 TCP | mxgw Galaxy browse (Q2) |
| vp03, vp04, all mxgw | nam1/nam2 | 636 TCP | NAM LDAPS |
| users, MES, Delmia | vp03, vp04 | 8085 (or 443 with TLS) | Central UI, Management, Inbound API |
| admins | all four | 5130 | mxgw dashboard |
| central and site | SMTP or EWS | 587 / 443 | Notifications, if in scope |
| Mac (deploy) | all four | 5985, 445 | WinRM deploy scope + SMB (only if Q6 allows) |
The LocalDb sync row is verified: the rig's site-a-node-a uses PeerAddress: http://scadabridge-site-a-b:8083,
the existing site gRPC listener.
5. Rollback
- Before Phase 7 step 5, prod is greenfield. Rollback = stop and disable the services. QA keeps serving every consumer.
- After consumers are cut over, rollback = point MES and Delmia back at QA, and redeploy the moved machines on
QA. The
.bakfolders and DB backup follow the same pattern as QA (binaries +COPY_ONLYbackup per window). - Every later prod upgrade uses the QA pack pattern: stop both sites, stop both centrals, migrate, swap, start the centrals, start the sites.
6. Risks
| Risk | Mitigation |
|---|---|
The central↔site boundary (8083) is blocked or filtered in one direction only. QA never exercised it, because it was co-located |
Phase 0 firewall ticket; Phase 6 round-trip gate in both directions |
| One machine driven by two stacks (Q9) | Undeploy from QA before the prod deploy |
_PROD groups grant wider access than intended |
T6 membership review before mapping |
| A service-account lockout breaks every app | A1P is no-lockout; stop after 2 service-account failures in 15 minutes (cutover runbook rule) |
| The mxgw upgrade script aborts on a slow start | Phase 1 step 1 fix |
| The security software flags remote exec | Q6. Batch the on-box steps into a few scripts per phase, and stop at the first alert |