dohertj2/scadaproj
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(inventory): add the QA stack (vq03/vq04/sql-vq03) and correct vd03 DEV-vs-prod

Browse Source 
deployment.md:
- §1/§2.1: add wonder-app-vq03, wonder-app-vq04 and wonder-sql-vq03 with IPs and
  per-host roles; new §7.1 QA port map + live active/standby state.
- QA is a real 2-node HA pair (ScadaBridge central+site pairs, OtOpcUa pair) where
  DEV is a single all-in-one box, so it is the only environment here that exercises
  failover and leader election. MxGateway/HistorianGateway/Overview run on vq03 only.
- Every QA fact is marked scan (port probe) or dash (read off the QA Overview
  dashboard's live polling). No QA config file was read; connection strings and
  service names stay unverified.
- Correct the tier: vd03 is DEV, not production (vd=dev, vq=qa, per
  mes-delmia-integration-api.md §1.1). Its ASPNETCORE_ENVIRONMENT really is
  "Production" -- a .NET overlay name, orthogonal to the tier, and the likely source
  of the original error. No production box has been identified.
- Record that QA requires Basic auth on /management where DEV does not, so DEV-only
  procedures relying on unauthenticated management calls will not work on QA.
- Record that inbound API keys never cross environments (per-stack peppered-HMAC,
  excluded from config bundles) plus the 401-vs-400 probe, with the caveat that auth
  runs before route resolution so a 401 needs a positive control.
- Withdraw the SSH-on-2222 claim made earlier in this same pass: all three wonder
  boxes accept TCP on 2222 and send no SSH banner, so servecli is not serving SSH and
  no shell access was available. Port scan cannot establish a service; banner grab can.

CLAUDE.md:
- Overview: record the second instance on wonder-app-vq03:5320, and that its real-FQDN
  registry is correct rather than a divergence -- the 127.0.0.1 rule is specific to
  vd03 co-locating every instance on one host.
- Add the DEV-vs-prod note pointing at deployment.md §2.1/§7.1.
This commit is contained in:
Joseph Doherty
2026-08-13 08:08:23 -04:00
parent 818775b311
commit f42aa14033
2 changed files with 129 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+13
View File
@@ -337,6 +337,19 @@ service `ZbOverview` on **:5320** under the shared NSSM, `ASPNETCORE_ENVIRONMENT
**site** card claim `Leader · central`, a different Akka cluster. Fixed by using `127.0.0.1`,
which matches nothing, so each chip falls back to the raw Akka address.
**A second instance runs on the QA box `wonder-app-vq03:5320`** (observed live 2026-08-13, watching
**8 instances across 4 applications**, all Up — it is the fastest way to read QA's whole topology).
Its registry uses **real FQDNs**, and that is correct rather than a divergence: the `127.0.0.1`
rule above exists only because VD03 co-locates every instance on one host. QA is a **2-node pair**
(`vq03`/`vq04`) with genuinely distinct hosts, so leader resolution works as designed and the chips
resolve to real names (`node-a (vq03)`). Carry the *reason*, not the literal `127.0.0.1`.
> ⚠️ **`wonder-app-vd03` is the DEV box, not production** (`vd` = DEV, `vq` = QA — confirmed by
> [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Its
> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production`, but that is a .NET config-overlay name and
> is orthogonal to the deployment tier. No production box has been identified. Full host/port
> inventory for both tiers: [`deployment.md`](deployment.md) §2.1 + §7.1.
## Per-project primary commands
Run these from inside each project directory (not from `scadaproj`).
deployment.md
+116 -15
View File
@@ -5,6 +5,11 @@
> actual compose/Traefik/SSH files (not docs alone). For the per-service **environment
> variables** see the companion [`env_vars.md`](env_vars.md).
>
> **Updated 2026-08-13:** added the **QA stack** (`wonder-app-vq03` / `wonder-app-vq04` /
> `wonder-sql-vq03` — [§7.1](#71-qa-port-map-vq03-and-vq04)), established by probing the live boxes.
> Same pass corrected `wonder-app-vd03`, which this doc had called "production" and which is the
> **DEV** box (see the ⚠️ note in [§2.1](#21-host-inventory)).
>
> **Source confidence:** container/port/Traefik/DB facts below are read straight from the
> compose + `traefik/*.yml` files. SSH facts are from `~/.ssh/config` + `~/.ssh/known_hosts`.
> Where a fact is referenced in repo docs but not pinned in a config/script on this machine,
@@ -21,7 +26,8 @@
| **ScadaBridge `infra`** | This Mac | Shared backing services (MSSQL, OPC-UA sims, SMTP, REST, Playwright) — **not** LDAP (see shared GLAuth below) | n/a (deps) |
| **OtOpcUa `otopcua-dev`** | This Mac | 3 independent Akka clusters (MAIN + SITE-A + SITE-B) sharing one ConfigDb | `http://localhost:9200` |
| **MxAccessGateway** | `windev` (10.100.0.48), Windows | Windows-native gRPC gateway + per-session x86 worker (no Docker) | `http://10.100.0.48:5120` (gRPC) |
| **Production (VD03)** | `wonder-app-vd03.zmr.zimmer.com` | Single-node ScadaBridge + MxGateway prod host | see `docs/operations/` runbooks |
| **DEV (VD03)** | `wonder-app-vd03.zmr.zimmer.com` | Single-node, all-in-one: ScadaBridge + OtOpcUa + MxGateway + HistorianGateway + Overview | see `docs/operations/` runbooks |
| **QA (VQ03/VQ04)** | `wonder-app-vq03` + `wonder-app-vq04` (+ `wonder-sql-vq03`) | **2-node HA pair** — ScadaBridge central+site pairs, OtOpcUa pair; singletons on VQ03 | `http://wonder-app-vq03.zmr.zimmer.com:5320` (Overview) |
The three ScadaBridge stacks share one external Docker network **`scadabridge-net`**; the
OtOpcUa `otopcua-dev` stack runs on its **own** default network (`otopcua-dev_default`) and is
@@ -42,10 +48,21 @@ collide (see [§7](#7-consolidated-host-port-map)).
| **This Mac** | local | macOS (darwin) | Dev workstation — runs all local Docker stacks | n/a |
| **windev** | `10.100.0.48` | Windows | OtOpcUa Windows-service host **+** MxAccessGateway (gRPC `5120` / dashboard `5130`) | 22 |
| **fixture host** | `10.100.0.35` | Debian/Linux + Docker | OtOpcUa driver **integration-test fixtures** + a test SQL Server | 22 |
| **VD03 (prod)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | Production single-node ScadaBridge + MxGateway | **2222** |
| **VD03 (DEV)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | **DEV** all-in-one box: ScadaBridge (single-node) + OtOpcUa + MxGateway + HistorianGateway + Overview | **2222** |
| **VQ03 (QA)** | `wonder-app-vq03.zmr.zimmer.com` (`10.220.158.2`) | Windows | **QA node A** — ScadaBridge central-a + site-a, OtOpcUa node-a, **plus the singleton services**: MxGateway, HistorianGateway, Overview | **2222** |
| **VQ04 (QA)** | `wonder-app-vq04.zmr.zimmer.com` (`10.220.158.3`) | Windows | **QA node B** — ScadaBridge central-b + site-b, OtOpcUa node-b. **No** MxGateway/HistorianGateway/Overview | **2222** |
| **wonder-sql-vq03** | `wonder-sql-vq03.zmr.zimmer.com` | Windows | **QA data tier** — SQL Server (`1433`) + AVEVA Historian gRPC TLS (`32565`); mirrors `wonder-sql-vd03` | _(not probed)_ |
| **gitea** | `gitea.dohertylan.com` (`10.100.0.228`) | Linux | Git remotes + NuGet feed (`/api/packages/dohertj2/nuget`) | 22 |
All are on the private `10.x` lab network — a LAN/VPN connection is required.
The lab hosts (`windev`, fixture, gitea) are on the private `10.x` network; the **wonder** hosts are
on the corporate `10.220.15x` network — both need the LAN/VPN.
> ⚠️ **`vd` = DEV, `vq` = QA.** VD03 was labelled "production" throughout this doc until 2026-08-13;
> it is not. The convention is confirmed by the family's own MES spec (DEV `wonder-sql-vd01` vs QA
> `wondersqlqa`, [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Note VD03's
> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production` — that is a .NET config-overlay name and
> says nothing about the deployment tier; the two axes are orthogonal. **No production box has been
> identified** — do not assume either of these is it.
### 2.2 How to connect (passwordless SSH)
@@ -67,9 +84,20 @@ Host windev
|---|---|---|
| **windev** (Win host) | `ssh windev` | Configured alias; user `dohertj2`, key `id_ed25519`, port 22 |
| **fixture host** | `ssh dohertj2@10.100.0.35` | In `known_hosts`; **no** config alias — pass user explicitly; port 22, key-based |
| **VD03 (prod)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (the only non-standard SSH port); user/key not pinned in config — confirm before use |
| **VD03 (DEV)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (servecli, not OpenSSH); user/key not pinned in config — confirm before use. Durable key `~/.ssh/servecli_wonder`. **servecli does not survive a reboot** — if it's down, use RDP |
| **VQ03 / VQ04 (QA)** | `ssh dohertj2@wonder-app-vq0{3,4}.zmr.zimmer.com -p 2222` | ⚠️ **Not working 2026-08-13** — see the banner note below |
| **local Linux VMs** | `ssh orb` / `orb` | OrbStack-managed |
> ⚠️ **A TCP-open `2222` does NOT mean SSH works.** On 2026-08-13 all three wonder boxes
> (VD03, VQ03, VQ04) accepted a TCP connection on `2222` and then sent **no SSH banner** — the
> server must send its identification string first (RFC 4253 §4.2), so a bannerless socket is a
> listening port with no working SSH service behind it. The visible symptom is `ssh` hanging until
> it is killed, *not* a refused connection or an auth error. Every session attempt that day failed
> this way, so **no shell access to any wonder box was available** and nothing could be deployed.
> Probe it with a banner grab, never a port scan:
> `exec 3<>/dev/tcp/<host>/2222 && head -c 60 <&3`.
> servecli is known not to survive a reboot; the documented fallback is RDP.
> ⚠️ `~/bin` is **empty** on this Mac. OtOpcUa's `CLAUDE.md` mentions an `lmxopcua-fix` helper "in
> `~/bin`" for controlling the `10.100.0.35` fixture containers — it is **not present here** (it's a
> Windows-side helper). On this machine, drive the fixture host with direct SSH, e.g.
@@ -253,8 +281,9 @@ dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86
- **Worker model:** the Server spawns one `ZB.MOM.WW.MxGateway.Worker.exe` (x86) **per gRPC session**;
IPC over a named pipe (`\\.\pipe\mxgateway-<session>` + a per-session `MXGATEWAY_WORKER_NONCE`);
heartbeat 5s / grace 15s; max 64 concurrent sessions. The worker exits when the session closes.
- **Production hosts:** both `10.100.0.48` and `wonder-app-vd03` serve gRPC on `:5120` (per
`docs/GatewayConfiguration.md`).
- **Deployed hosts:** `10.100.0.48` (windev), `wonder-app-vd03` (DEV) and `wonder-app-vq03` (QA)
all serve gRPC on `:5120` (per `docs/GatewayConfiguration.md`; the VQ03 port was probed
2026-08-13). **QA runs MxGateway on VQ03 only** — VQ04 has no gateway.
### 5.3 Who connects to it
@@ -320,7 +349,58 @@ Every published host port across the local stacks (no collisions — all can run
| 8089 | `otopcua-dev` traefik :8080 | Traefik dashboard | otopcua-dev |
**Remote (non-local) endpoints:** MxAccessGateway gRPC `10.100.0.48:5120` (h2c) / dashboard `:5130`;
production gRPC on `wonder-app-vd03:5120`. SSH: windev/fixture/gitea on `22`, **VD03 on `2222`**.
DEV gRPC on `wonder-app-vd03:5120`, QA on `wonder-app-vq03:5120`. SSH: windev/fixture/gitea on `22`,
**VD03/VQ03/VQ04 on `2222`**.
### 7.1 QA port map: VQ03 and VQ04
Established **2026-08-13** from two independent live sources, neither of them a config file — treat
connection strings and service names as unverified. Rows are marked with how each was established:
- **`scan`** — TCP port scan of both hosts, plus a `/health/ready` GET where one was served.
- **`dash`** — read off the **QA Overview dashboard** (`vq03:5320`), which was already polling the
QA fleet and reports each instance's endpoint, health entries and active/standby state. Stronger
evidence than a scan for these rows: the dashboard is successfully talking to the service, not
merely finding an open socket. These ports were never scanned by me.
| Port | VQ03 | VQ04 | How | Service |
|---|---|---|---|---|
| 9000 | ✅ | ✅ | dash | **OtOpcUa** host + AdminUI (same listener) |
| 4840 | ✅ | ✅ | scan | OtOpcUa **OPC UA** endpoint (`opc.tcp`) |
| 4053 | ✅ | ✅ | dash | OtOpcUa Akka remoting (from the reported leader address) |
| 8085 | ✅ | ✅ | scan | **ScadaBridge Central** — CentralUI + Management + Inbound API |
| 8084 | ✅ | ✅ | dash | ScadaBridge **Site** health listener (no UI; `/` → 404) |
| 8081 | ✅ | ✅ | scan | ScadaBridge Central Akka remoting |
| 8082 / 8083 | ✅ | ✅ | scan | ScadaBridge Site Akka / gRPC control plane (h2c — rejects an HTTP/1.1 GET) |
| 5120 / 5130 | ✅ | ✗ | scan | **MxAccessGateway** gRPC / dashboard |
| 5220 / 5221 | ✅ | ✗ | scan | **HistorianGateway** dashboard / gRPC h2c |
| 5320 | ✅ | ✗ | scan | **Overview** dashboard (the QA fleet's own single pane) |
| 1433 | ✅ | ✗ | scan | SQL Server |
| 2222 | ⚠️ | ⚠️ | scan | servecli SSH — **TCP-open but NOT serving SSH** (see below) |
**Live state at capture** (`8 instances across 4 applications`, all Up):
| Application | VQ03 | VQ04 |
|---|---|---|
| OtOpcUa (cluster `MAIN`) | node-a — **Standby** (`active 503`) | node-b — **Active** (`active 200`) |
| ScadaBridge `central` | central-a — **Active** | central-b — Standby |
| ScadaBridge `site` | site-a — **Active** | site-b — Standby |
| MxAccessGateway | Active | — |
| HistorianGateway | Up | — |
Three things worth carrying forward:
1. **QA is a real 2-node HA pair; DEV (VD03) is a single box.** So QA is the only environment here
where failover, leader election and the active/standby split are actually exercised. A change
that works on VD03 has not been tested against a pair.
2. **The active node is not the same for every application.** At capture, ScadaBridge's central and
site singletons were on VQ03 while OtOpcUa's active node was VQ04 — these are independent Akka
clusters and there is no reason for them to agree. Don't infer one from the other.
3. **QA's Overview registry can use real FQDNs, unlike VD03's.** `LeaderResolver` matches a reported
Akka leader to an instance **by host only**, which needs distinct hosts per instance — true on QA,
false on the co-located VD03 box, where the workaround is to point every `BaseUrl` at `127.0.0.1`
so resolution never matches and each chip falls back to the raw Akka address. On QA the leader
chips resolve to real names (`node-a (vq03)`), so that workaround is **not** wanted here.
---
@@ -328,14 +408,35 @@ production gRPC on `wonder-app-vd03:5120`. SSH: windev/fixture/gitea on `22`, **
Every credential shown above (`OtOpcUa!Dev123`, `ScadaBridge_Dev1#`, the inline API-key peppers,
the `docker-dev` JWT signing key, the `mxgw_…` API key) is a **dev-only placeholder** for the local
stacks — never reuse as a real secret. Production injects real secrets out-of-band (NSSM env / secret
store), per ScadaBridge `docs/operations/inbound-api-key-reissue.md` (the VD03 runbook). The full
stacks — never reuse as a real secret. The deployed wonder boxes (DEV + QA) inject real secrets
out-of-band (NSSM env / secret store), per ScadaBridge
`docs/operations/inbound-api-key-reissue.md`. Secrets are **per-stack** — a DEV value is never a QA
value, and inbound API keys in particular cannot be copied between them ([§9](#9-wonder-boxes-dev-and-qa)). The full
🔒 secret inventory and the `__`-env-var override forms are in [`env_vars.md`](env_vars.md) §5.
## 9. Production (VD03) — pointer
## 9. Wonder boxes: DEV and QA
`wonder-app-vd03.zmr.zimmer.com` (SSH `:2222`) runs the production single-node ScadaBridge and the
MxGateway (gRPC `:5120`). The production install is **not a scripted in-repo flow** here — the
operational procedures live in ScadaBridge `docs/operations/` (`failover-procedures.md`,
`maintenance-procedures.md`, `inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any
prod service/port specifics not in those runbooks as unverified.
`wonder-app-vd03.zmr.zimmer.com` (SSH `:2222`) is the **DEV** all-in-one box — single-node
ScadaBridge plus OtOpcUa, MxGateway (gRPC `:5120`), HistorianGateway and Overview.
`wonder-app-vq03` + `wonder-app-vq04` are the **QA** pair ([§7.1](#71-qa-port-map-vq03-and-vq04)).
Neither install is a scripted in-repo flow — the operational procedures live in ScadaBridge
`docs/operations/` (`failover-procedures.md`, `maintenance-procedures.md`,
`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in
those runbooks, or not probed in §7.1, as unverified.
**Auth posture differs between the two, and QA is the correct one:**
| | VD03 (DEV) | VQ03/VQ04 (QA) |
|---|---|---|
| `POST :8085/management` | **No auth** (`DisableLogin=true`) — `ListApiKeys`/`CreateApiKey` succeed unauthenticated from off-box | **Basic auth required** (`AUTH_FAILED` without it) |
That is expected on a dev box and is **not** a finding there, but it means any DEV-only procedure
that relies on unauthenticated management calls **will not work on QA** — supply credentials.
**Inbound API keys do not cross environments.** Each stack mints its own secrets (peppered-HMAC,
never recoverable) and config bundles deliberately carry no keys, so a QA `sbk_…` token 401s on DEV
and its keyId is absent from DEV's store entirely — verified 2026-08-13. Probe recipe: `POST
:8085/api/MesMoveIn -d '{}'` → **`400 VALIDATION_FAILED`** means the key is valid on that stack
(it reached the method and executed nothing); `401` means rejected. Pair it with a positive control
— auth runs *before* route resolution, so a bad **method name** also returns `401`.