scadaproj/deployment.md

Deployment & Environments — SCADA/OT family

How the sister projects are deployed: environments, hosts, SSH access, Docker/Traefik topology, databases, and the full service/port map. Compiled 2026-06-03 by reading the actual compose/Traefik/SSH files (not docs alone). For the per-service environment variables see the companion env_vars.md.

Updated 2026-08-13: added the QA stack (wonder-app-vq03 / wonder-app-vq04 / wonder-sql-vq03 — §7.1), established by probing the live boxes. Same pass corrected wonder-app-vd03, which this doc had called "production" and which is the DEV box (see the ⚠️ note in §2.1).

Source confidence: container/port/Traefik/DB facts below are read straight from the compose + traefik/*.yml files. SSH facts are from ~/.ssh/config + ~/.ssh/known_hosts. Where a fact is referenced in repo docs but not pinned in a config/script on this machine, it's marked (referenced, not scripted in-repo) — don't treat those as automated.

---

1. Environment inventory

Environment	Where it runs	What it is	Entry point
ScadaBridge docker	This Mac (Docker Desktop/OrbStack)	Full hub-and-spoke: 2 Central + 3 sites ×2 nodes	http://localhost:9000
ScadaBridge docker-env2	This Mac	Second isolated cluster: 2 Central + 1 site ×2 nodes	http://localhost:9100
ScadaBridge infra	This Mac	Shared backing services (MSSQL, OPC-UA sims, SMTP, REST, Playwright) — not LDAP (see shared GLAuth below)	n/a (deps)
OtOpcUa otopcua-dev	This Mac	3 independent Akka clusters (MAIN + SITE-A + SITE-B) sharing one ConfigDb	http://localhost:9200
MxAccessGateway	windev (10.100.0.48), Windows	Windows-native gRPC gateway + per-session x86 worker (no Docker)	http://10.100.0.48:5120 (gRPC)
DEV (VD03)	wonder-app-vd03.zmr.zimmer.com	Single-node, all-in-one: ScadaBridge + OtOpcUa + MxGateway + HistorianGateway + Overview	see docs/operations/ runbooks
QA (VQ03/VQ04)	wonder-app-vq03 + wonder-app-vq04 (+ wonder-sql-vq03)	2-node HA pair — ScadaBridge central+site pairs, OtOpcUa pair; singletons on VQ03	http://wonder-app-vq03.zmr.zimmer.com:5320 (Overview)

The three ScadaBridge stacks share one external Docker network scadabridge-net; the OtOpcUa otopcua-dev stack runs on its own default network (otopcua-dev_default) and is network-isolated from ScadaBridge. All local stacks can run simultaneously — host ports do not collide (see §7).

On this Apple-Silicon Mac, MSSQL runs under amd64 emulation (slow first-ready; the "platform does not match" warning is expected/benign). See scadabridge-local-deploy-gotchas.

---

2. Hosts & SSH connectivity

2.1 Host inventory

Host	Address	OS	Role	SSH port
This Mac	local	macOS (darwin)	Dev workstation — runs all local Docker stacks	n/a
windev	10.100.0.48	Windows	OtOpcUa Windows-service host + MxAccessGateway (gRPC 5120 / dashboard 5130)	22
fixture host	10.100.0.35	Debian/Linux + Docker	OtOpcUa driver integration-test fixtures + a test SQL Server	22
VD03 (DEV)	wonder-app-vd03.zmr.zimmer.com	Windows	DEV all-in-one box: ScadaBridge (single-node) + OtOpcUa + MxGateway + HistorianGateway + Overview	2222
VQ03 (QA)	wonder-app-vq03.zmr.zimmer.com (10.220.158.2)	Windows	QA node A — ScadaBridge central-a + site-a, OtOpcUa node-a, plus the singleton services: MxGateway, HistorianGateway, Overview	2222
VQ04 (QA)	wonder-app-vq04.zmr.zimmer.com (10.220.158.3)	Windows	QA node B — ScadaBridge central-b + site-b, OtOpcUa node-b. No MxGateway/HistorianGateway/Overview	2222
wonder-sql-vq03	wonder-sql-vq03.zmr.zimmer.com	Windows	QA data tier — SQL Server (1433) + AVEVA Historian gRPC TLS (32565); mirrors wonder-sql-vd03	(not probed)
gitea	gitea.dohertylan.com (10.100.0.228)	Linux	Git remotes + NuGet feed (/api/packages/dohertj2/nuget)	22

The lab hosts (windev, fixture, gitea) are on the private 10.x network; the wonder hosts are on the corporate 10.220.15x network — both need the LAN/VPN.

⚠️ vd = DEV, vq = QA. VD03 was labelled "production" throughout this doc until 2026-08-13; it is not. The convention is confirmed by the family's own MES spec (DEV wonder-sql-vd01 vs QA wondersqlqa, mes-delmia-integration-api.md §1.1). Note VD03's ASPNETCORE_ENVIRONMENT genuinely is Production — that is a .NET config-overlay name and says nothing about the deployment tier; the two axes are orthogonal. No production box has been identified — do not assume either of these is it.

2.2 How to connect (passwordless SSH)

Auth is key-based (passwordless) with ~/.ssh/id_ed25519 (a legacy ~/.ssh/id_rsa exists as fallback). Only one host alias is defined in ~/.ssh/config:

# ~/.ssh/config (verified)
Include ~/.orbstack/ssh/config        # OrbStack local Linux VMs — use `ssh orb` / `orb` CLI

Host windev
    HostName 10.100.0.48
    User dohertj2
    IdentityFile ~/.ssh/id_ed25519
    # Port 22 (default)

Target	Command	Notes
windev (Win host)	ssh windev	Configured alias; user dohertj2, key id_ed25519, port 22
fixture host	ssh dohertj2@10.100.0.35	In known_hosts; no config alias — pass user explicitly; port 22, key-based
VD03 (DEV)	ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222	In known_hosts on port 2222 (servecli, not OpenSSH); user/key not pinned in config — confirm before use. Durable key ~/.ssh/servecli_wonder. servecli does not survive a reboot — if it's down, use RDP
VQ03 / VQ04 (QA)	ssh dohertj2@wonder-app-vq0{3,4}.zmr.zimmer.com -p 2222	⚠️ Not working 2026-08-13 — see the banner note below
local Linux VMs	ssh orb / orb	OrbStack-managed

⚠️ A TCP-open 2222 does NOT mean SSH works. On 2026-08-13 all three wonder boxes (VD03, VQ03, VQ04) accepted a TCP connection on 2222 and then sent no SSH banner — the server must send its identification string first (RFC 4253 §4.2), so a bannerless socket is a listening port with no working SSH service behind it. The visible symptom is ssh hanging until it is killed, not a refused connection or an auth error. Every session attempt that day failed this way, so no shell access to any wonder box was available and nothing could be deployed. Probe it with a banner grab, never a port scan: exec 3<>/dev/tcp/<host>/2222 && head -c 60 <&3. servecli is known not to survive a reboot; the documented fallback is RDP.

⚠️ ~/bin is empty on this Mac. OtOpcUa's CLAUDE.md mentions an lmxopcua-fix helper "in ~/bin" for controlling the 10.100.0.35 fixture containers — it is not present here (it's a Windows-side helper). On this machine, drive the fixture host with direct SSH, e.g. ssh dohertj2@10.100.0.35 'docker compose -f /opt/otopcua-<driver>/docker-compose.yml up -d'. Treat the exact remote paths/commands as (referenced, not scripted in-repo) — verify on the host.

---

3. ScadaBridge deployment

.NET 10 + Akka.NET. One image scadabridge:latest (built by docker/build.sh) backs every node; role is chosen by SCADABRIDGE_CONFIG (Central|Site) → appsettings.{role}.json. Central is a 2-node Akka cluster (split-brain resolver = keep-oldest); each Site is its own 2-node Akka cluster reached from Central via ClusterClient.

3.1 docker/ — primary 3-site cluster (network scadabridge-net)

Service	Container	Host→container ports	Role	Volumes
central-a	scadabridge-central-a	9001:5000 (UI+Inbound API), 9011:8081 (Akka)	Central	central-node-a/appsettings.Central.json (ro), …/logs
central-b	scadabridge-central-b	9002:5000, 9012:8081	Central	central-node-b/…
site-a-a	scadabridge-site-a-a	9021:8082 (Akka), 9023:8083 (gRPC)	Site	site-a-node-a/{appsettings.Site.json,data,logs}
site-a-b	scadabridge-site-a-b	9022:8082, 9024:8083	Site	site-a-node-b/…
site-b-a	scadabridge-site-b-a	9031:8082, 9033:8083	Site	site-b-node-a/…
site-b-b	scadabridge-site-b-b	9032:8082, 9034:8083	Site	site-b-node-b/…
site-c-a	scadabridge-site-c-a	9041:8082, 9043:8083	Site	site-c-node-a/…
site-c-b	scadabridge-site-c-b	9042:8082, 9044:8083	Site	site-c-node-b/…
traefik	scadabridge-traefik	9000:80 (Central LB), 8180:8080 (dashboard)	LB	traefik/{traefik,dynamic}.yml (ro)

All restart: unless-stopped; image scadabridge:latest (traefik traefik:v3.4). Access: Central UI/API via LB http://localhost:9000; direct nodes :9001/:9002; Traefik dashboard http://localhost:8180; Management API http://localhost:9000/management; health …/health/ready + …/health/active.

3.2 docker-env2/ — secondary 1-site cluster (same scadabridge-net)

Service	Container	Host→container ports	Role
central-a	scadabridge-env2-central-a	9101:5000, 9111:8081	Central
central-b	scadabridge-env2-central-b	9102:5000, 9112:8081	Central
site-x-a	scadabridge-env2-site-x-a	9121:8082, 9123:8083	Site
site-x-b	scadabridge-env2-site-x-b	9122:8082, 9124:8083	Site
traefik	scadabridge-env2-traefik	9100:80 (LB), 8181:8080 (dashboard)	LB

Access: LB http://localhost:9100; direct :9101/:9102; dashboard http://localhost:8181. This cluster's DBs and auth cookie name are distinct from docker/ so the two can run on localhost at once — cookie ZB.MOM.WW.ScadaBridge.Auth.env2 vs the default; see scadabridge-local-deploy-gotchas.

3.3 infra/ — shared backing services (network scadabridge-net)

Service	Container	Image	Host ports	Purpose
mssql	scadabridge-mssql	mcr.microsoft.com/mssql/server:2022-latest	1433:1433	SQL Server — Central DBs for both clusters; named vol scadabridge-mssql-data; init via /docker-entrypoint-initdb.d/{setup,machinedata_seed,setup-env2}.sql
opcua	scadabridge-opcua	mcr.microsoft.com/iotedge/opc-plc:latest	50000:50000, 8080:8080	OPC-UA simulator 1 (--unsecuretransport --autoaccept)
opcua2	scadabridge-opcua2	…/opc-plc:latest	50010:50010, 8081:8080	OPC-UA simulator 2
smtp	scadabridge-smtp	axllent/mailpit:latest	1025:1025, 8025:8025	SMTP sink + web UI (http://localhost:8025)
restapi	scadabridge-restapi	local build ./restapi	5200:5200	Test REST endpoint
playwright	scadabridge-playwright	mcr.microsoft.com/playwright:v1.58.2-noble	3000:3000	Browser-automation server

LDAP is NOT started by infra/. The per-app scadabridge-ldap container has been retired (commented out in infra/docker-compose.yml). All three apps (ScadaBridge, OtOpcUa, MxAccessGateway) now share a single zb-shared-glauth container on the Linux fixture host 10.100.0.35:3893 (baseDN dc=zb,dc=local, Transport=None). Source of truth and deploy/verify runbook: scadaproj/infra/glauth/ (config.toml + docker-compose.yml + README.md); deploy by scp-ing those two files to 10.100.0.35 and running docker compose up -d.

3.4 Traefik (ScadaBridge)

Both clusters use a file provider + insecure API dashboard. traefik.yml: entrypoint web:80, api.dashboard: true / insecure: true, file provider dynamic.yml. dynamic.yml router central (PathPrefix(/) → service central) load-balances the two Central containers with an active health check on /health/active (interval 5s, timeout 3s) — so traffic only routes to the active leader (standby returns 503 and is dropped from rotation):

# docker/traefik/dynamic.yml  (env2 points at scadabridge-env2-central-a/-b)
http:
  routers:   { central: { rule: "PathPrefix(`/`)", service: central, entryPoints: [web] } }
  services:
    central:
      loadBalancer:
        healthCheck: { path: /health/active, interval: 5s, timeout: 3s }
        servers: [ {url: "http://scadabridge-central-a:5000"}, {url: "http://scadabridge-central-b:5000"} ]

3.5 Databases (ScadaBridge)

• Central → MSSQL (scadabridge-mssql:1433), app login scadabridge_app / ScadaBridge_Dev1# 🔒(dev-only):
  • docker/: ScadaBridgeConfig + ScadaBridgeMachineData
  • docker-env2/: ScadaBridgeConfig2 + ScadaBridgeMachineData2
  • Created by infra/mssql/setup.sql + setup-env2.sql at MSSQL init; EF Core migrations run on Central startup; docker-env2/init-db.sh ensures the env2 DBs before deploy; seed-sites.sh seeds Site rows post-deploy.
• Site → SQLite, per node under the mounted …/data volume (SiteDbPath, plus a store-and-forward DB). Not networked, not replicated across hosts.

3.6 Deploy commands (ScadaBridge)

cd ~/Desktop/ScadaBridge
cd infra && docker compose up -d        # 1) backing services (MSSQL, OPC-UA, SMTP, REST) — LDAP is shared glauth on 10.100.0.35 (scadaproj/infra/glauth/)
bash docker/build.sh                     # 2) create scadabridge-net (if missing) + build scadabridge:latest
bash docker/deploy.sh                    # 3) up -d --force-recreate; prints access points (9000/9001/9002/8180)
bash docker/seed-sites.sh                # 4) seed sites + data-connections (optional)
# env2 cluster:
bash docker-env2/deploy.sh               #   reuses the image; runs init-db.sh; ports 9100/9101/9102/8181

Caveat: deploy.sh does up -d --force-recreate, starting both Central nodes at once — they can split-brain on a simultaneous start. Start Central sequenced (central-a → wait /health/active 200 → central-b). Central also requires ScadaBridge__InboundApi__ApiKeyPepper (dev value is inline in both composes). Full detail: scadabridge-local-deploy-gotchas.

---

4. OtOpcUa deployment (otopcua-dev)

.NET 10 OPC-UA server. Three independent Akka clusters share the single OtOpcUa ConfigDb (multi-tenancy via the ServerCluster table); Akka isolation is by disjoint seed lists (same system name otopcua, internal remoting port 4053). Built locally from docker-dev/Dockerfile → image otopcua-host:dev. No per-app LDAP container — docker-dev is un-stubbed (Authentication__Ldap__DevStubMode removed) and binds the shared GLAuth at 10.100.0.35:3893 (baseDN dc=zb,dc=local, Transport=None). Start the shared glauth first via scadaproj/infra/glauth/ if it is not already running.

Service	Container	Host→container ports	Cluster / role
sql	(otopcua-dev-sql-1)	14330:1433	SQL Server 2022 — the shared OtOpcUa ConfigDb
cluster-seed	one-shot	—	mssql-tools running /seed/entrypoint.sh (idempotent ServerCluster/ClusterNode seed)
admin-a	host	(none — internal :9000 UI behind Traefik)	MAIN, role admin (seed)
admin-b	host	(none)	MAIN, role admin (joins admin-a)
driver-a	host	4840:4840 (OPC UA)	MAIN, role driver
driver-b	host	4841:4840	MAIN, role driver
site-a-1	host	4842:4840	SITE-A, admin,driver (seed)
site-a-2	host	4843:4840	SITE-A, admin,driver
site-b-1	host	4844:4840	SITE-B, admin,driver (seed)
site-b-2	host	4845:4840	SITE-B, admin,driver
traefik	host	9200:80 (Admin UI LB), 8089:8080 (dashboard)	traefik:v3.1

• OPC UA endpoints: opc.tcp://localhost:4840 (driver-a) … :4845 (site-b-2). Admin nodes serve no OPC UA.
• Admin UI (Traefik, sticky cookie otopcua_lb, health-checked on /health/active):
  • MAIN cluster: http://localhost:9200
  • SITE-A: http://site-a.localhost:9200 · SITE-B: http://site-b.localhost:9200 (Host-header routing; macOS auto-resolves *.localhost)
  • Traefik dashboard: http://localhost:8089
• DB: sql service, 14330:1433, SA OtOpcUa!Dev123 🔒(dev-only), database OtOpcUa; EF auto-migrates on host start, then cluster-seed inserts the 3 ServerCluster + 6 ClusterNode rows.
• Deploy: docker compose -f docker-dev/docker-compose.yml up -d --build ; tear down with … down -v.
• Galaxy link: driver nodes resolve GALAXY_MXGW_API_KEY and connect out to MxAccessGateway (see §5).

Integration-test fixtures (separate from this stack) run on the Linux fixture host 10.100.0.35 (Modbus :5020, Allen-Bradley :44818, S7 :102, OPC-UA :50000, SQL :14330). Those are test endpoints, not the deployed app; per-fixture env defaults are in env_vars.md §1.3.

---

5. MxAccessGateway deployment (Windows-native, no Docker)

Two processes: an x64 .NET 10 Server (ASP.NET Core gRPC + Blazor dashboard) and a per-session x86 .NET 4.8 Worker that owns the 32-bit AVEVA MXAccess COM/STA. Windows-only. Deployed on windev (10.100.0.48) and VD03, run as a Windows Service via NSSM (config delivered as Kestrel__Endpoints__… environment variables, not appsettings.json).

5.1 Endpoint/port map

Endpoint	Default URL	Protocol	Config key	Purpose
Http (gRPC)	http://0.0.0.0:5120 (h2c)	HTTP/2 cleartext	Kestrel__Endpoints__Http__Url / __Protocols=Http2	Public gRPC: sessions, MxCommand/MxEvent, Galaxy browse
Dashboard	http://0.0.0.0:5130	HTTP/1.1	Kestrel__Endpoints__Dashboard__Url	Blazor dashboard + SignalR hubs + /login

Local dev (launchSettings.json): gRPC http://localhost:5120 (https dev profile adds 7121). TLS optional — set …Http__Url=https://…; the gateway auto-generates a self-signed cert if none is supplied (docs/GatewayConfiguration.md). Dashboard cookie name is now configurable (MxGateway:Dashboard:CookieName).

5.2 Run / host

# local dev
dotnet run --project src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj
# the x86 worker must be published first; path = MxGateway:Worker:ExecutablePath
dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86

• Worker model: the Server spawns one ZB.MOM.WW.MxGateway.Worker.exe (x86) per gRPC session; IPC over a named pipe (\\.\pipe\mxgateway-<session> + a per-session MXGATEWAY_WORKER_NONCE); heartbeat 5s / grace 15s; max 64 concurrent sessions. The worker exits when the session closes.
• Deployed hosts: 10.100.0.48 (windev), wonder-app-vd03 (DEV) and wonder-app-vq03 (QA) all serve gRPC on :5120 (per docs/GatewayConfiguration.md; the VQ03 port was probed 2026-08-13). QA runs MxGateway on VQ03 only — VQ04 has no gateway.

5.3 Who connects to it

Client	Connects to	Auth
OtOpcUa GalaxyDriver	http://10.100.0.48:5120 (gRPC)	API key via GALAXY_MXGW_API_KEY (mxgw_… bearer) 🔒
ScadaBridge MxGateway adapter	same gRPC endpoint :5120	API key

---

6. Cross-project runtime data flow (deployed)

AVEVA Galaxy (Wonderware)  ──MXAccess COM (32-bit)──►  MxAccessGateway  (windev:5120 gRPC / :5130 dashboard)
                                                              ▲           ▲
                              OtOpcUa GalaxyDriver ───gRPC────┘           │ gRPC
                              (otopcua-dev: opc.tcp :4840–4845)           │
                                     │ OPC UA                             │
                                     ▼                                    │
                              ScadaBridge DCL  ◄──OPC UA──┐   ┌──MxGateway adapter──┘
                              (docker :9000 / env2 :9100) └───┘

ScadaBridge reaches Wonderware data two ways: (1) OPC UA → OtOpcUa → gateway, or (2) its MxGateway adapter → gateway directly. The break surface is the wire contracts (the gateway .protos and OtOpcUa's OPC-UA address space), not compile references.

---

7. Consolidated host port map

Every published host port across the local stacks (no collisions — all can run at once):

Port	→ Container:port	Service	Stack
1025	scadabridge-smtp:1025	SMTP submission	infra
1433	scadabridge-mssql:1433	SQL Server (ScadaBridge Central DBs)	infra
3000	scadabridge-playwright:3000	Playwright server	infra
3893	zb-shared-glauth:3893 on 10.100.0.35	LDAP (shared GLAuth — remote fixture host, not a local container)	scadaproj/infra/glauth/
5200	scadabridge-restapi:5200	Test REST API	infra
8025	scadabridge-smtp:8025	Mailpit web UI	infra
8080	scadabridge-opcua:8080	OPC-UA sim 1 web UI	infra
8081	scadabridge-opcua2:8080	OPC-UA sim 2 web UI	infra
50000	scadabridge-opcua:50000	OPC-UA sim 1 endpoint	infra
50010	scadabridge-opcua2:50010	OPC-UA sim 2 endpoint	infra
9000	scadabridge-traefik:80	Central UI/API (LB)	docker
8180	scadabridge-traefik:8080	Traefik dashboard	docker
9001 / 9002	central-a / central-b :5000	Central UI+Inbound API (direct)	docker
9011 / 9012	central-a / central-b :8081	Akka remoting	docker
9021–9024	site-a-a/b :8082 / :8083	Site A Akka / gRPC	docker
9031–9034	site-b-a/b :8082 / :8083	Site B Akka / gRPC	docker
9041–9044	site-c-a/b :8082 / :8083	Site C Akka / gRPC	docker
9100	scadabridge-env2-traefik:80	Central UI/API (LB)	docker-env2
8181	scadabridge-env2-traefik:8080	Traefik dashboard	docker-env2
9101 / 9102	env2 central-a / central-b :5000	Central (direct)	docker-env2
9111 / 9112	env2 central-a / central-b :8081	Akka remoting	docker-env2
9121–9124	env2 site-x-a/b :8082 / :8083	Site X Akka / gRPC	docker-env2
14330	otopcua-dev sql :1433	SQL Server (OtOpcUa DB)	otopcua-dev
4840 / 4841	driver-a / driver-b :4840	OPC UA (MAIN)	otopcua-dev
4842 / 4843	site-a-1 / site-a-2 :4840	OPC UA (SITE-A)	otopcua-dev
4844 / 4845	site-b-1 / site-b-2 :4840	OPC UA (SITE-B)	otopcua-dev
9200	otopcua-dev traefik :80	Admin UI (LB)	otopcua-dev
8089	otopcua-dev traefik :8080	Traefik dashboard	otopcua-dev

Remote (non-local) endpoints: MxAccessGateway gRPC 10.100.0.48:5120 (h2c) / dashboard :5130; DEV gRPC on wonder-app-vd03:5120, QA on wonder-app-vq03:5120. SSH: windev/fixture/gitea on 22, VD03/VQ03/VQ04 on 2222.

7.1 QA port map: VQ03 and VQ04

Established 2026-08-13 from two independent live sources, neither of them a config file — treat connection strings and service names as unverified. Rows are marked with how each was established:

• scan — TCP port scan of both hosts, plus a /health/ready GET where one was served.
• dash — read off the QA Overview dashboard (vq03:5320), which was already polling the QA fleet and reports each instance's endpoint, health entries and active/standby state. Stronger evidence than a scan for these rows: the dashboard is successfully talking to the service, not merely finding an open socket. These ports were never scanned by me.

Port	VQ03	VQ04	How	Service
9000	✅	✅	dash	OtOpcUa host + AdminUI (same listener)
4840	✅	✅	scan	OtOpcUa OPC UA endpoint (opc.tcp)
4053	✅	✅	dash	OtOpcUa Akka remoting (from the reported leader address)
8085	✅	✅	scan	ScadaBridge Central — CentralUI + Management + Inbound API
8084	✅	✅	dash	ScadaBridge Site health listener (no UI; / → 404)
8081	✅	✅	scan	ScadaBridge Central Akka remoting
8082 / 8083	✅	✅	scan	ScadaBridge Site Akka / gRPC control plane (h2c — rejects an HTTP/1.1 GET)
5120 / 5130	✅	✗	scan	MxAccessGateway gRPC / dashboard
5220 / 5221	✅	✗	scan	HistorianGateway dashboard / gRPC h2c
5320	✅	✗	scan	Overview dashboard (the QA fleet's own single pane)
1433	✅	✗	scan	SQL Server
2222	⚠️	⚠️	scan	servecli SSH — TCP-open but NOT serving SSH (see below)

Live state at capture (8 instances across 4 applications, all Up):

Application	VQ03	VQ04
OtOpcUa (cluster MAIN)	node-a — Standby (active 503)	node-b — Active (active 200)
ScadaBridge central	central-a — Active	central-b — Standby
ScadaBridge site	site-a — Active	site-b — Standby
MxAccessGateway	Active	—
HistorianGateway	Up	—

Three things worth carrying forward:

1. QA is a real 2-node HA pair; DEV (VD03) is a single box. So QA is the only environment here where failover, leader election and the active/standby split are actually exercised. A change that works on VD03 has not been tested against a pair.
2. The active node is not the same for every application. At capture, ScadaBridge's central and site singletons were on VQ03 while OtOpcUa's active node was VQ04 — these are independent Akka clusters and there is no reason for them to agree. Don't infer one from the other.
3. QA's Overview registry can use real FQDNs, unlike VD03's. LeaderResolver matches a reported Akka leader to an instance by host only, which needs distinct hosts per instance — true on QA, false on the co-located VD03 box, where the workaround is to point every BaseUrl at 127.0.0.1 so resolution never matches and each chip falls back to the raw Akka address. On QA the leader chips resolve to real names (node-a (vq03)), so that workaround is not wanted here.

---

8. Secrets & dev-only values

Every credential shown above (OtOpcUa!Dev123, ScadaBridge_Dev1#, the inline API-key peppers, the docker-dev JWT signing key, the mxgw_… API key) is a dev-only placeholder for the local stacks — never reuse as a real secret. The deployed wonder boxes (DEV + QA) inject real secrets out-of-band (NSSM env / secret store), per ScadaBridge docs/operations/inbound-api-key-reissue.md. Secrets are per-stack — a DEV value is never a QA value, and inbound API keys in particular cannot be copied between them (§9). The full 🔒 secret inventory and the __-env-var override forms are in env_vars.md §5.

9. Wonder boxes: DEV and QA

wonder-app-vd03.zmr.zimmer.com (SSH :2222) is the DEV all-in-one box — single-node ScadaBridge plus OtOpcUa, MxGateway (gRPC :5120), HistorianGateway and Overview. wonder-app-vq03 + wonder-app-vq04 are the QA pair (§7.1).

Neither install is a scripted in-repo flow — the operational procedures live in ScadaBridge docs/operations/ (failover-procedures.md, maintenance-procedures.md, inbound-api-key-reissue.md, troubleshooting-guide.md). Treat any service/port specifics not in those runbooks, or not probed in §7.1, as unverified.

Auth posture differs between the two, and QA is the correct one:

	VD03 (DEV)	VQ03/VQ04 (QA)
POST :8085/management	No auth (DisableLogin=true) — ListApiKeys/CreateApiKey succeed unauthenticated from off-box	Basic auth required (AUTH_FAILED without it)

That is expected on a dev box and is not a finding there, but it means any DEV-only procedure that relies on unauthenticated management calls will not work on QA — supply credentials.

Inbound API keys do not cross environments. Each stack mints its own secrets (peppered-HMAC, never recoverable) and config bundles deliberately carry no keys, so a QA sbk_… token 401s on DEV and its keyId is absent from DEV's store entirely — verified 2026-08-13. Probe recipe: POST :8085/api/MesMoveIn -d '{}' → 400 VALIDATION_FAILED means the key is valid on that stack (it reached the method and executed nothing); 401 means rejected. Pair it with a positive control — auth runs before route resolution, so a bad method name also returns 401.