scadaproj/deployment.md

# Deployment & Environments — SCADA/OT family

> How the sister projects are deployed: environments, hosts, SSH access, Docker/Traefik
> topology, databases, and the full service/port map. Compiled **2026-06-03** by reading the
> actual compose/Traefik/SSH files (not docs alone). For the per-service **environment
> variables** see the companion [`env_vars.md`](env_vars.md).
>
> **Updated 2026-08-13:** added the **QA stack** (`wonder-app-vq03` / `wonder-app-vq04` /
> `wonder-sql-vq03` — [§7.1](#71-qa-port-map-vq03-and-vq04)), established by probing the live boxes.
> Same pass corrected `wonder-app-vd03`, which this doc had called "production" and which is the
> **DEV** box (see the ⚠️ note in [§2.1](#21-host-inventory)).
>
> **Source confidence:** container/port/Traefik/DB facts below are read straight from the
> compose + `traefik/*.yml` files. SSH facts are from `~/.ssh/config` + `~/.ssh/known_hosts`.
> Where a fact is referenced in repo docs but not pinned in a config/script on this machine,
> it's marked _(referenced, not scripted in-repo)_ — don't treat those as automated.

---

## 1. Environment inventory

| Environment | Where it runs | What it is | Entry point |
|---|---|---|---|
| **ScadaBridge `docker`** | This Mac (Docker Desktop/OrbStack) | Full hub-and-spoke: 2 Central + 3 sites ×2 nodes | `http://localhost:9000` |
| **ScadaBridge `docker-env2`** | This Mac | Second isolated cluster: 2 Central + 1 site ×2 nodes | `http://localhost:9100` |
| **ScadaBridge `infra`** | This Mac | Shared backing services (MSSQL, OPC-UA sims, SMTP, REST, Playwright) — **not** LDAP (see shared GLAuth below) | n/a (deps) |
| **OtOpcUa `otopcua-dev`** | This Mac | 3 independent Akka clusters (MAIN + SITE-A + SITE-B) sharing one ConfigDb | `http://localhost:9200` |
| **MxAccessGateway** | `windev` (10.100.0.48), Windows | Windows-native gRPC gateway + per-session x86 worker (no Docker) | `http://10.100.0.48:5120` (gRPC) |
| **DEV (VD03)** | `wonder-app-vd03.zmr.zimmer.com` | Single-node, all-in-one: ScadaBridge + OtOpcUa + MxGateway + HistorianGateway + Overview | see `docs/operations/` runbooks |
| **QA (VQ03/VQ04)** | `wonder-app-vq03` + `wonder-app-vq04` (+ `wonder-sql-vq03`) | **2-node HA pair** — ScadaBridge central+site pairs, OtOpcUa pair; singletons on VQ03 | `http://wonder-app-vq03.zmr.zimmer.com:5320` (Overview) |

The three ScadaBridge stacks share one external Docker network **`scadabridge-net`**; the
OtOpcUa `otopcua-dev` stack runs on its **own** default network (`otopcua-dev_default`) and is
network-isolated from ScadaBridge. All local stacks can run simultaneously — host ports do not
collide (see [§7](#7-consolidated-host-port-map)).

> On this Apple-Silicon Mac, MSSQL runs under amd64 emulation (slow first-ready; the "platform
> does not match" warning is expected/benign). See [[scadabridge-local-deploy-gotchas]].

---

## 2. Hosts & SSH connectivity

### 2.1 Host inventory

| Host | Address | OS | Role | SSH port |
|---|---|---|---|---|
| **This Mac** | local | macOS (darwin) | Dev workstation — runs all local Docker stacks | n/a |
| **windev** | `10.100.0.48` | Windows | OtOpcUa Windows-service host **+** MxAccessGateway (gRPC `5120` / dashboard `5130`) | 22 |
| **fixture host** | `10.100.0.35` | Debian/Linux + Docker | OtOpcUa driver **integration-test fixtures** + a test SQL Server | 22 |
| **VD03 (DEV)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | **DEV** all-in-one box: ScadaBridge (single-node) + OtOpcUa + MxGateway + HistorianGateway + Overview | **2222** |
| **VQ03 (QA)** | `wonder-app-vq03.zmr.zimmer.com` (`10.220.158.2`) | Windows | **QA node A** — ScadaBridge central-a + site-a, OtOpcUa node-a, **plus the singleton services**: MxGateway, HistorianGateway, Overview | **2222** |
| **VQ04 (QA)** | `wonder-app-vq04.zmr.zimmer.com` (`10.220.158.3`) | Windows | **QA node B** — ScadaBridge central-b + site-b, OtOpcUa node-b, **plus (since 2026-08-14) its own MxGateway, HistorianGateway and Overview** (cloned from VQ03 incl. API-key stores; Galaxy SQL points at VQ03 — no local SQL Server) | **2222** |
| **wonder-sql-vq03** | `wonder-sql-vq03.zmr.zimmer.com` | Windows | **QA data tier** — SQL Server (`1433`) + AVEVA Historian gRPC TLS (`32565`); mirrors `wonder-sql-vd03` | _(not probed)_ |
| **gitea** | `gitea.dohertylan.com` (`10.100.0.228`) | Linux | Git remotes + NuGet feed (`/api/packages/dohertj2/nuget`) | 22 |

The lab hosts (`windev`, fixture, gitea) are on the private `10.x` network; the **wonder** hosts are
on the corporate `10.220.15x` network — both need the LAN/VPN.

> ⚠️ **`vd` = DEV, `vq` = QA.** VD03 was labelled "production" throughout this doc until 2026-08-13;
> it is not. The convention is confirmed by the family's own MES spec (DEV `wonder-sql-vd01` vs QA
> `wondersqlqa`, [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Note VD03's
> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production` — that is a .NET config-overlay name and
> says nothing about the deployment tier; the two axes are orthogonal. **No production box has been
> identified** — do not assume either of these is it.

### 2.2 How to connect (passwordless SSH)

Auth is **key-based (passwordless)** with `~/.ssh/id_ed25519` (a legacy `~/.ssh/id_rsa` exists
as fallback). Only **one** host alias is defined in `~/.ssh/config`:

```sshconfig
# ~/.ssh/config (verified)
Include ~/.orbstack/ssh/config        # OrbStack local Linux VMs — use `ssh orb` / `orb` CLI

Host windev
    HostName 10.100.0.48
    User dohertj2
    IdentityFile ~/.ssh/id_ed25519
    # Port 22 (default)
```

| Target | Command | Notes |
|---|---|---|
| **windev** (Win host) | `ssh windev` | Configured alias; user `dohertj2`, key `id_ed25519`, port 22 |
| **fixture host** | `ssh dohertj2@10.100.0.35` | In `known_hosts`; **no** config alias — pass user explicitly; port 22, key-based |
| **VD03 (DEV)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (servecli, not OpenSSH); user/key not pinned in config — confirm before use. Durable key `~/.ssh/servecli_wonder`. **servecli does not survive a reboot** — if it's down, use RDP |
| **VQ03 / VQ04 (QA)** | `psbridge --server wonder-app-vq0{3,4}.zmr.zimmer.com:6150 <cmd>` | **The supported path since 2026-08-14** (servecli `2222` is retired on these boxes — see the banner note below). Root jail `E:\psrem`; see the `psbridge` skill |
| **local Linux VMs** | `ssh orb` / `orb` | OrbStack-managed |

> ⚠️ **A TCP-open `2222` does NOT mean SSH works.** On 2026-08-13 all three wonder boxes
> (VD03, VQ03, VQ04) accepted a TCP connection on `2222` and then sent **no SSH banner** — the
> server must send its identification string first (RFC 4253 §4.2), so a bannerless socket is a
> listening port with no working SSH service behind it. The visible symptom is `ssh` hanging until
> it is killed, *not* a refused connection or an auth error. Every session attempt that day failed
> this way, so **no shell access to any wonder box was available** and nothing could be deployed.
> Probe it with a banner grab, never a port scan:
> `exec 3<>/dev/tcp/<host>/2222 && head -c 60 <&3`.
> servecli is known not to survive a reboot; the documented fallback is RDP.

> ⚠️ `~/bin` is **empty** on this Mac. OtOpcUa's `CLAUDE.md` mentions an `lmxopcua-fix` helper "in
> `~/bin`" for controlling the `10.100.0.35` fixture containers — it is **not present here** (it's a
> Windows-side helper). On this machine, drive the fixture host with direct SSH, e.g.
> `ssh dohertj2@10.100.0.35 'docker compose -f /opt/otopcua-<driver>/docker-compose.yml up -d'`.
> Treat the exact remote paths/commands as _(referenced, not scripted in-repo)_ — verify on the host.

---

## 3. ScadaBridge deployment

.NET 10 + Akka.NET. One image `scadabridge:latest` (built by `docker/build.sh`) backs every node;
role is chosen by `SCADABRIDGE_CONFIG` (`Central`|`Site`) → `appsettings.{role}.json`. Central is a
2-node Akka cluster (split-brain resolver = `keep-oldest`); each Site is its **own** 2-node Akka
cluster reached from Central via ClusterClient.

### 3.1 `docker/` — primary 3-site cluster (network `scadabridge-net`)

| Service | Container | Host→container ports | Role | Volumes |
|---|---|---|---|---|
| central-a | `scadabridge-central-a` | `9001:5000` (UI+Inbound API), `9011:8081` (Akka) | Central | `central-node-a/appsettings.Central.json` (ro), `…/logs` |
| central-b | `scadabridge-central-b` | `9002:5000`, `9012:8081` | Central | `central-node-b/…` |
| site-a-a | `scadabridge-site-a-a` | `9021:8082` (Akka), `9023:8083` (gRPC) | Site | `site-a-node-a/{appsettings.Site.json,data,logs}` |
| site-a-b | `scadabridge-site-a-b` | `9022:8082`, `9024:8083` | Site | `site-a-node-b/…` |
| site-b-a | `scadabridge-site-b-a` | `9031:8082`, `9033:8083` | Site | `site-b-node-a/…` |
| site-b-b | `scadabridge-site-b-b` | `9032:8082`, `9034:8083` | Site | `site-b-node-b/…` |
| site-c-a | `scadabridge-site-c-a` | `9041:8082`, `9043:8083` | Site | `site-c-node-a/…` |
| site-c-b | `scadabridge-site-c-b` | `9042:8082`, `9044:8083` | Site | `site-c-node-b/…` |
| traefik | `scadabridge-traefik` | `9000:80` (Central LB), `8180:8080` (dashboard) | LB | `traefik/{traefik,dynamic}.yml` (ro) |

All `restart: unless-stopped`; image `scadabridge:latest` (traefik `traefik:v3.4`).
**Access:** Central UI/API via LB `http://localhost:9000`; direct nodes `:9001`/`:9002`; Traefik
dashboard `http://localhost:8180`; Management API `http://localhost:9000/management`; health
`…/health/ready` + `…/health/active`.

### 3.2 `docker-env2/` — secondary 1-site cluster (same `scadabridge-net`)

| Service | Container | Host→container ports | Role |
|---|---|---|---|
| central-a | `scadabridge-env2-central-a` | `9101:5000`, `9111:8081` | Central |
| central-b | `scadabridge-env2-central-b` | `9102:5000`, `9112:8081` | Central |
| site-x-a | `scadabridge-env2-site-x-a` | `9121:8082`, `9123:8083` | Site |
| site-x-b | `scadabridge-env2-site-x-b` | `9122:8082`, `9124:8083` | Site |
| traefik | `scadabridge-env2-traefik` | `9100:80` (LB), `8181:8080` (dashboard) | LB |

**Access:** LB `http://localhost:9100`; direct `:9101`/`:9102`; dashboard `http://localhost:8181`.
This cluster's DBs and **auth cookie name** are distinct from `docker/` so the two can run on
`localhost` at once — cookie `ZB.MOM.WW.ScadaBridge.Auth.env2` vs the default; see
[[scadabridge-local-deploy-gotchas]].

### 3.3 `infra/` — shared backing services (network `scadabridge-net`)

| Service | Container | Image | Host ports | Purpose |
|---|---|---|---|---|
| mssql | `scadabridge-mssql` | `mcr.microsoft.com/mssql/server:2022-latest` | `1433:1433` | SQL Server — Central DBs for **both** clusters; named vol `scadabridge-mssql-data`; init via `/docker-entrypoint-initdb.d/{setup,machinedata_seed,setup-env2}.sql` |
| opcua | `scadabridge-opcua` | `mcr.microsoft.com/iotedge/opc-plc:latest` | `50000:50000`, `8080:8080` | OPC-UA simulator 1 (`--unsecuretransport --autoaccept`) |
| opcua2 | `scadabridge-opcua2` | `…/opc-plc:latest` | `50010:50010`, `8081:8080` | OPC-UA simulator 2 |
| smtp | `scadabridge-smtp` | `axllent/mailpit:latest` | `1025:1025`, `8025:8025` | SMTP sink + web UI (`http://localhost:8025`) |
| restapi | `scadabridge-restapi` | local build `./restapi` | `5200:5200` | Test REST endpoint |
| playwright | `scadabridge-playwright` | `mcr.microsoft.com/playwright:v1.58.2-noble` | `3000:3000` | Browser-automation server |

> **LDAP is NOT started by `infra/`.** The per-app `scadabridge-ldap` container has been retired
> (commented out in `infra/docker-compose.yml`). All three apps (ScadaBridge, OtOpcUa, MxAccessGateway)
> now share a single **`zb-shared-glauth`** container on the Linux fixture host **`10.100.0.35:3893`**
> (`baseDN dc=zb,dc=local`, Transport=None). Source of truth and deploy/verify runbook:
> **`scadaproj/infra/glauth/`** (`config.toml` + `docker-compose.yml` + `README.md`); deploy by
> scp-ing those two files to `10.100.0.35` and running `docker compose up -d`.

### 3.4 Traefik (ScadaBridge)

Both clusters use a file provider + insecure API dashboard. `traefik.yml`: entrypoint `web:80`,
`api.dashboard: true / insecure: true`, file provider `dynamic.yml`. `dynamic.yml` router
`central` (`PathPrefix(/)` → service `central`) load-balances the two Central containers with an
**active health check** on `/health/active` (interval 5s, timeout 3s) — so traffic only routes to
the active leader (standby returns 503 and is dropped from rotation):

```yaml
# docker/traefik/dynamic.yml  (env2 points at scadabridge-env2-central-a/-b)
http:
  routers:   { central: { rule: "PathPrefix(`/`)", service: central, entryPoints: [web] } }
  services:
    central:
      loadBalancer:
        healthCheck: { path: /health/active, interval: 5s, timeout: 3s }
        servers: [ {url: "http://scadabridge-central-a:5000"}, {url: "http://scadabridge-central-b:5000"} ]
```

### 3.5 Databases (ScadaBridge)

- **Central → MSSQL** (`scadabridge-mssql:1433`), app login `scadabridge_app` / `ScadaBridge_Dev1#` 🔒(dev-only):
  - `docker/`: `ScadaBridgeConfig` + `ScadaBridgeMachineData`
  - `docker-env2/`: `ScadaBridgeConfig2` + `ScadaBridgeMachineData2`
  - Created by `infra/mssql/setup.sql` + `setup-env2.sql` at MSSQL init; EF Core migrations run on Central startup; `docker-env2/init-db.sh` ensures the env2 DBs before deploy; `seed-sites.sh` seeds Site rows post-deploy.
- **Site → SQLite**, per node under the mounted `…/data` volume (`SiteDbPath`, plus a store-and-forward DB). Not networked, not replicated across hosts.

### 3.6 Deploy commands (ScadaBridge)

```bash
cd ~/Desktop/ScadaBridge
cd infra && docker compose up -d        # 1) backing services (MSSQL, OPC-UA, SMTP, REST) — LDAP is shared glauth on 10.100.0.35 (scadaproj/infra/glauth/)
bash docker/build.sh                     # 2) create scadabridge-net (if missing) + build scadabridge:latest
bash docker/deploy.sh                    # 3) up -d --force-recreate; prints access points (9000/9001/9002/8180)
bash docker/seed-sites.sh                # 4) seed sites + data-connections (optional)
# env2 cluster:
bash docker-env2/deploy.sh               #   reuses the image; runs init-db.sh; ports 9100/9101/9102/8181
```

> **Caveat:** `deploy.sh` does `up -d --force-recreate`, starting both Central nodes at once — they
> can split-brain on a simultaneous start. Start Central **sequenced** (central-a → wait `/health/active`
> 200 → central-b). Central also requires `ScadaBridge__InboundApi__ApiKeyPepper` (dev value is inline in
> both composes). Full detail: [[scadabridge-local-deploy-gotchas]].

---

## 4. OtOpcUa deployment (`otopcua-dev`)

.NET 10 OPC-UA server. **Three independent Akka clusters** share the single `OtOpcUa` ConfigDb
(multi-tenancy via the `ServerCluster` table); Akka isolation is by disjoint seed lists (same
system name `otopcua`, internal remoting port `4053`). Built locally from `docker-dev/Dockerfile`
→ image `otopcua-host:dev`. **No per-app LDAP container** — `docker-dev` is un-stubbed
(`Authentication__Ldap__DevStubMode` removed) and binds the **shared GLAuth** at
`10.100.0.35:3893` (`baseDN dc=zb,dc=local`, Transport=None). Start the shared glauth first via
`scadaproj/infra/glauth/` if it is not already running.

| Service | Container | Host→container ports | Cluster / role |
|---|---|---|---|
| sql | (`otopcua-dev-sql-1`) | `14330:1433` | SQL Server 2022 — the shared `OtOpcUa` ConfigDb |
| cluster-seed | one-shot | — | `mssql-tools` running `/seed/entrypoint.sh` (idempotent ServerCluster/ClusterNode seed) |
| admin-a | host | _(none — internal `:9000` UI behind Traefik)_ | MAIN, role `admin` (seed) |
| admin-b | host | _(none)_ | MAIN, role `admin` (joins admin-a) |
| driver-a | host | `4840:4840` (OPC UA) | MAIN, role `driver` |
| driver-b | host | `4841:4840` | MAIN, role `driver` |
| site-a-1 | host | `4842:4840` | SITE-A, `admin,driver` (seed) |
| site-a-2 | host | `4843:4840` | SITE-A, `admin,driver` |
| site-b-1 | host | `4844:4840` | SITE-B, `admin,driver` (seed) |
| site-b-2 | host | `4845:4840` | SITE-B, `admin,driver` |
| traefik | host | `9200:80` (Admin UI LB), `8089:8080` (dashboard) | `traefik:v3.1` |

- **OPC UA endpoints:** `opc.tcp://localhost:4840` (driver-a) … `:4845` (site-b-2). Admin nodes serve no OPC UA.
- **Admin UI (Traefik, sticky cookie `otopcua_lb`, health-checked on `/health/active`):**
  - MAIN cluster: `http://localhost:9200`
  - SITE-A: `http://site-a.localhost:9200`  ·  SITE-B: `http://site-b.localhost:9200` (Host-header routing; macOS auto-resolves `*.localhost`)
  - Traefik dashboard: `http://localhost:8089`
- **DB:** `sql` service, `14330:1433`, SA `OtOpcUa!Dev123` 🔒(dev-only), database `OtOpcUa`; EF auto-migrates on host start, then `cluster-seed` inserts the 3 ServerCluster + 6 ClusterNode rows.
- **Deploy:** `docker compose -f docker-dev/docker-compose.yml up -d --build` ; tear down with `… down -v`.
- **Galaxy link:** driver nodes resolve `GALAXY_MXGW_API_KEY` and connect out to MxAccessGateway (see §5).

> **Integration-test fixtures (separate from this stack)** run on the Linux **fixture host
> `10.100.0.35`** (Modbus `:5020`, Allen-Bradley `:44818`, S7 `:102`, OPC-UA `:50000`, SQL `:14330`).
> Those are test endpoints, not the deployed app; per-fixture env defaults are in [`env_vars.md`](env_vars.md) §1.3.

---

## 5. MxAccessGateway deployment (Windows-native, no Docker)

Two processes: an **x64 .NET 10 Server** (ASP.NET Core gRPC + Blazor dashboard) and a **per-session
x86 .NET 4.8 Worker** that owns the 32-bit AVEVA MXAccess COM/STA. Windows-only. Deployed on
**`windev` (10.100.0.48)** and **VD03**, run as a **Windows Service via NSSM** (config delivered as
`Kestrel__Endpoints__…` environment variables, not `appsettings.json`).

### 5.1 Endpoint/port map

| Endpoint | Default URL | Protocol | Config key | Purpose |
|---|---|---|---|---|
| **Http (gRPC)** | `http://0.0.0.0:5120` (h2c) | HTTP/2 cleartext | `Kestrel__Endpoints__Http__Url` / `__Protocols=Http2` | Public gRPC: sessions, MxCommand/MxEvent, Galaxy browse |
| **Dashboard** | `http://0.0.0.0:5130` | HTTP/1.1 | `Kestrel__Endpoints__Dashboard__Url` | Blazor dashboard + SignalR hubs + `/login` |

Local dev (`launchSettings.json`): gRPC `http://localhost:5120` (https dev profile adds `7121`).
TLS optional — set `…Http__Url=https://…`; the gateway auto-generates a self-signed cert if none is
supplied (`docs/GatewayConfiguration.md`). Dashboard cookie name is now configurable
(`MxGateway:Dashboard:CookieName`).

### 5.2 Run / host

```powershell
# local dev
dotnet run --project src/ZB.MOM.WW.MxGateway.Server/ZB.MOM.WW.MxGateway.Server.csproj
# the x86 worker must be published first; path = MxGateway:Worker:ExecutablePath
dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86
```

- **Worker model:** the Server spawns one `ZB.MOM.WW.MxGateway.Worker.exe` (x86) **per gRPC session**;
  IPC over a named pipe (`\\.\pipe\mxgateway-<session>` + a per-session `MXGATEWAY_WORKER_NONCE`);
  heartbeat 5s / grace 15s; max 64 concurrent sessions. The worker exits when the session closes.
- **Deployed hosts:** `10.100.0.48` (windev), `wonder-app-vd03` (DEV), `wonder-app-vq03` and
  (since 2026-08-14) `wonder-app-vq04` (QA) all serve gRPC on `:5120` (per
  `docs/GatewayConfiguration.md`). **QA runs MxGateway on BOTH nodes** — VQ04's was cloned from
  VQ03 (same API-key store + pepper, so the same tokens verify on either; its Galaxy SQL and its
  alarm subscription `\\wonder-app-vq03\Galaxy!NAM` both point at VQ03).

### 5.3 Who connects to it

| Client | Connects to | Auth |
|---|---|---|
| OtOpcUa `GalaxyDriver` | `http://10.100.0.48:5120` (gRPC) | API key via `GALAXY_MXGW_API_KEY` (`mxgw_…` bearer) 🔒 |
| ScadaBridge MxGateway adapter | same gRPC endpoint `:5120` | API key |

---

## 6. Cross-project runtime data flow (deployed)

```
AVEVA Galaxy (Wonderware)  ──MXAccess COM (32-bit)──►  MxAccessGateway  (windev:5120 gRPC / :5130 dashboard)
                                                              ▲           ▲
                              OtOpcUa GalaxyDriver ───gRPC────┘           │ gRPC
                              (otopcua-dev: opc.tcp :4840–4845)           │
                                     │ OPC UA                             │
                                     ▼                                    │
                              ScadaBridge DCL  ◄──OPC UA──┐   ┌──MxGateway adapter──┘
                              (docker :9000 / env2 :9100) └───┘
```

ScadaBridge reaches Wonderware data two ways: **(1)** OPC UA → OtOpcUa → gateway, or **(2)** its
MxGateway adapter → gateway directly. The break surface is the wire contracts (the gateway `.proto`s
and OtOpcUa's OPC-UA address space), not compile references.

---

## 7. Consolidated host port map

Every published host port across the local stacks (no collisions — all can run at once):

| Port | → Container:port | Service | Stack |
|---|---|---|---|
| 1025 | `scadabridge-smtp`:1025 | SMTP submission | infra |
| 1433 | `scadabridge-mssql`:1433 | SQL Server (ScadaBridge Central DBs) | infra |
| 3000 | `scadabridge-playwright`:3000 | Playwright server | infra |
| 3893 | `zb-shared-glauth`:3893 on **10.100.0.35** | LDAP (shared GLAuth — remote fixture host, not a local container) | scadaproj/infra/glauth/ |
| 5200 | `scadabridge-restapi`:5200 | Test REST API | infra |
| 8025 | `scadabridge-smtp`:8025 | Mailpit web UI | infra |
| 8080 | `scadabridge-opcua`:8080 | OPC-UA sim 1 web UI | infra |
| 8081 | `scadabridge-opcua2`:8080 | OPC-UA sim 2 web UI | infra |
| 50000 | `scadabridge-opcua`:50000 | OPC-UA sim 1 endpoint | infra |
| 50010 | `scadabridge-opcua2`:50010 | OPC-UA sim 2 endpoint | infra |
| 9000 | `scadabridge-traefik`:80 | **Central UI/API (LB)** | docker |
| 8180 | `scadabridge-traefik`:8080 | Traefik dashboard | docker |
| 9001 / 9002 | central-a / central-b :5000 | Central UI+Inbound API (direct) | docker |
| 9011 / 9012 | central-a / central-b :8081 | Akka remoting | docker |
| 9021–9024 | site-a-a/b :8082 / :8083 | Site A Akka / gRPC | docker |
| 9031–9034 | site-b-a/b :8082 / :8083 | Site B Akka / gRPC | docker |
| 9041–9044 | site-c-a/b :8082 / :8083 | Site C Akka / gRPC | docker |
| 9100 | `scadabridge-env2-traefik`:80 | **Central UI/API (LB)** | docker-env2 |
| 8181 | `scadabridge-env2-traefik`:8080 | Traefik dashboard | docker-env2 |
| 9101 / 9102 | env2 central-a / central-b :5000 | Central (direct) | docker-env2 |
| 9111 / 9112 | env2 central-a / central-b :8081 | Akka remoting | docker-env2 |
| 9121–9124 | env2 site-x-a/b :8082 / :8083 | Site X Akka / gRPC | docker-env2 |
| 14330 | `otopcua-dev` sql :1433 | SQL Server (`OtOpcUa` DB) | otopcua-dev |
| 4840 / 4841 | driver-a / driver-b :4840 | OPC UA (MAIN) | otopcua-dev |
| 4842 / 4843 | site-a-1 / site-a-2 :4840 | OPC UA (SITE-A) | otopcua-dev |
| 4844 / 4845 | site-b-1 / site-b-2 :4840 | OPC UA (SITE-B) | otopcua-dev |
| 9200 | `otopcua-dev` traefik :80 | **Admin UI (LB)** | otopcua-dev |
| 8089 | `otopcua-dev` traefik :8080 | Traefik dashboard | otopcua-dev |

**Remote (non-local) endpoints:** MxAccessGateway gRPC `10.100.0.48:5120` (h2c) / dashboard `:5130`;
DEV gRPC on `wonder-app-vd03:5120`, QA on `wonder-app-vq03:5120`. SSH: windev/fixture/gitea on `22`,
**VD03/VQ03/VQ04 on `2222`**.

### 7.1 QA port map: VQ03 and VQ04

Established **2026-08-13** from two independent live sources, neither of them a config file, and
**updated 2026-08-14** when the three singleton services were cloned onto VQ04 (that update IS from
the boxes' live NSSM config + health probes, so its rows are verified). Rows are marked with how
each was established:

- **`scan`** — TCP port scan of both hosts, plus a `/health/ready` GET where one was served.
- **`dash`** — read off the **QA Overview dashboard** (`vq03:5320`), which was already polling the
  QA fleet and reports each instance's endpoint, health entries and active/standby state. Stronger
  evidence than a scan for these rows: the dashboard is successfully talking to the service, not
  merely finding an open socket. These ports were never scanned by me.
- **`svc` / `live`** — added 2026-08-14: read from the boxes' own service registry (NSSM
  `AppEnvironmentExtra` / `Win32_Service`) or exercised end-to-end during the VQ04 singleton setup.

| Port | VQ03 | VQ04 | How | Service |
|---|---|---|---|---|
| 9000 | ✅ | ✅ | dash | **OtOpcUa** host + AdminUI (same listener) |
| 4840 | ✅ | ✅ | scan | OtOpcUa **OPC UA** endpoint (`opc.tcp`) |
| 4053 | ✅ | ✅ | dash | OtOpcUa Akka remoting (from the reported leader address) |
| 8085 | ✅ | ✅ | scan | **ScadaBridge Central** — CentralUI + Management + Inbound API |
| 8084 | ✅ | ✅ | dash | ScadaBridge **Site** health listener (no UI; `/` → 404) |
| 8081 | ✅ | ✅ | scan | ScadaBridge Central Akka remoting |
| 8082 / 8083 | ✅ | ✅ | scan | ScadaBridge Site Akka / gRPC control plane (h2c — rejects an HTTP/1.1 GET) |
| 5120 / 5130 | ✅ | ✅ | scan | **MxAccessGateway** gRPC / dashboard (VQ04 added 2026-08-14, cloned from VQ03) |
| 5220 / 5221 | ✅ | ✅ | scan | **HistorianGateway** dashboard / gRPC h2c (VQ04 added 2026-08-14) |
| 5320 | ✅ | ✅ | scan | **Overview** dashboard (one per node since 2026-08-14; both watch the same 10-instance registry) |
| 1433 | ✅ | ✗ | scan | SQL Server (**ZB Galaxy repo DB — VQ03 only**; VQ04's gateways point their Galaxy connection strings here) |
| 3893 | ✅ | ✅ | svc | glauth LDAP (`ScadaBridge-LDAP` service, loopback-only, `dc=scadabridge,dc=local` — one per box, configs identical) |
| **6150** | ✅ | ✅ | live | **psbridge** remote-admin bridge (`server.exe`, root jail `E:\psrem`) — **the supported access path**; see §2.1 |
| 2222 | ⚠️ | ⚠️ | scan | servecli SSH — **retired; TCP-open but NOT serving SSH** (see below) |

**Live state at the 2026-08-13 capture** (`8 instances across 4 applications`, all Up; since
2026-08-14 the registry is **10 instances** — MxAccessGateway and HistorianGateway run on both
nodes, verified Healthy at setup):

| Application | VQ03 | VQ04 |
|---|---|---|
| OtOpcUa (cluster `MAIN`) | node-a — **Standby** (`active 503`) | node-b — **Active** (`active 200`) |
| ScadaBridge `central` | central-a — **Active** | central-b — Standby |
| ScadaBridge `site` | site-a — **Active** | site-b — Standby |
| MxAccessGateway | Active | Active (since 2026-08-14; independent gateways, no cluster) |
| HistorianGateway | Up | Up (since 2026-08-14) |
| Overview | Up (`:5320`) | Up (`:5320`, since 2026-08-14; same 10-instance registry) |

**The 2026-08-14 VQ04 clone, in one paragraph:** MxAccessGw + HistorianGateway + ZbOverview were
copied from VQ03 (same `E:\ApiInstall\...` layout, NSSM `AppEnvironmentExtra` config, LocalSystem)
**including the API-key stores and peppers, so the same `mxgw_…`/`histgw_…` tokens verify on both
boxes** — proven with positive + negative controls. The only config deltas: both apps' Galaxy SQL
connection strings point at VQ03 (`Server=wonder-app-vq03…;Database=ZB` — VQ04 has no local SQL
Server) and mxgw's alarm subscription stays `\\wonder-app-vq03\Galaxy!NAM`. **So VQ04's gateways
depend on VQ03 for Galaxy browse + alarms**; Historian/Runtime-DB traffic goes to `wonder-sql-vq03`
independently. Deploy record: `E:\ApiInstall\_DEPLOY-NOTES-20260814-singletons-added.txt` on VQ04.

Three things worth carrying forward:

1. **QA is a real 2-node HA pair; DEV (VD03) is a single box.** So QA is the only environment here
   where failover, leader election and the active/standby split are actually exercised. A change
   that works on VD03 has not been tested against a pair.
2. **The active node is not the same for every application.** At capture, ScadaBridge's central and
   site singletons were on VQ03 while OtOpcUa's active node was VQ04 — these are independent Akka
   clusters and there is no reason for them to agree. Don't infer one from the other.
3. **QA's Overview registry can use real FQDNs, unlike VD03's.** `LeaderResolver` matches a reported
   Akka leader to an instance **by host only**, which needs distinct hosts per instance — true on QA,
   false on the co-located VD03 box, where the workaround is to point every `BaseUrl` at `127.0.0.1`
   so resolution never matches and each chip falls back to the raw Akka address. On QA the leader
   chips resolve to real names (`node-a (vq03)`), so that workaround is **not** wanted here.

---

## 8. Secrets & dev-only values

Every credential shown above (`OtOpcUa!Dev123`, `ScadaBridge_Dev1#`, the inline API-key peppers,
the `docker-dev` JWT signing key, the `mxgw_…` API key) is a **dev-only placeholder** for the local
stacks — never reuse as a real secret. The deployed wonder boxes (DEV + QA) inject real secrets
out-of-band (NSSM env / secret store), per ScadaBridge
`docs/operations/inbound-api-key-reissue.md`. Secrets are **per-stack** — a DEV value is never a QA
value, and inbound API keys in particular cannot be copied between them ([§9](#9-wonder-boxes-dev-and-qa)). The full
🔒 secret inventory and the `__`-env-var override forms are in [`env_vars.md`](env_vars.md) §5.

## 9. Wonder boxes: DEV and QA

`wonder-app-vd03.zmr.zimmer.com` (SSH `:2222`) is the **DEV** all-in-one box — single-node
ScadaBridge plus OtOpcUa, MxGateway (gRPC `:5120`), HistorianGateway and Overview.
`wonder-app-vq03` + `wonder-app-vq04` are the **QA** pair ([§7.1](#71-qa-port-map-vq03-and-vq04)).

Neither install is a scripted in-repo flow — the operational procedures live in ScadaBridge
`docs/operations/` (`failover-procedures.md`, `maintenance-procedures.md`,
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
those runbooks, or not probed in §7.1, as unverified.

**Auth posture differs between the two, and QA is the correct one:**

| | VD03 (DEV) | VQ03/VQ04 (QA) |
|---|---|---|
| `POST :8085/management` | **No auth** (`DisableLogin=true`) — `ListApiKeys`/`CreateApiKey` succeed unauthenticated from off-box | **Basic auth required** (`AUTH_FAILED` without it) |

That is expected on a dev box and is **not** a finding there, but it means any DEV-only procedure
that relies on unauthenticated management calls **will not work on QA** — supply credentials.

**Inbound API keys do not cross environments.** Each stack mints its own secrets (peppered-HMAC,
never recoverable) and config bundles deliberately carry no keys, so a QA `sbk_…` token 401s on DEV
and its keyId is absent from DEV's store entirely — verified 2026-08-13. Probe recipe: `POST
:8085/api/MesMoveIn -d '{}'` → **`400 VALIDATION_FAILED`** means the key is valid on that stack
(it reached the method and executed nothing); `401` means rejected. Pair it with a positive control
— auth runs *before* route resolution, so a bad **method name** also returns `401`.