diff --git a/CLAUDE.md b/CLAUDE.md index c7b59a8..5758aa7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -337,6 +337,19 @@ service `ZbOverview` on **:5320** under the shared NSSM, `ASPNETCORE_ENVIRONMENT **site** card claim `Leader · central`, a different Akka cluster. Fixed by using `127.0.0.1`, which matches nothing, so each chip falls back to the raw Akka address. +**A second instance runs on the QA box `wonder-app-vq03:5320`** (observed live 2026-08-13, watching +**8 instances across 4 applications**, all Up — it is the fastest way to read QA's whole topology). +Its registry uses **real FQDNs**, and that is correct rather than a divergence: the `127.0.0.1` +rule above exists only because VD03 co-locates every instance on one host. QA is a **2-node pair** +(`vq03`/`vq04`) with genuinely distinct hosts, so leader resolution works as designed and the chips +resolve to real names (`node-a (vq03)`). Carry the *reason*, not the literal `127.0.0.1`. + +> ⚠️ **`wonder-app-vd03` is the DEV box, not production** (`vd` = DEV, `vq` = QA — confirmed by +> [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Its +> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production`, but that is a .NET config-overlay name and +> is orthogonal to the deployment tier. No production box has been identified. Full host/port +> inventory for both tiers: [`deployment.md`](deployment.md) §2.1 + §7.1. + ## Per-project primary commands Run these from inside each project directory (not from `scadaproj`). diff --git a/deployment.md b/deployment.md index b6cd8c7..25edafc 100644 --- a/deployment.md +++ b/deployment.md @@ -5,6 +5,11 @@ > actual compose/Traefik/SSH files (not docs alone). For the per-service **environment > variables** see the companion [`env_vars.md`](env_vars.md). > +> **Updated 2026-08-13:** added the **QA stack** (`wonder-app-vq03` / `wonder-app-vq04` / +> `wonder-sql-vq03` — [§7.1](#71-qa-port-map-vq03-and-vq04)), established by probing the live boxes. +> Same pass corrected `wonder-app-vd03`, which this doc had called "production" and which is the +> **DEV** box (see the ⚠️ note in [§2.1](#21-host-inventory)). +> > **Source confidence:** container/port/Traefik/DB facts below are read straight from the > compose + `traefik/*.yml` files. SSH facts are from `~/.ssh/config` + `~/.ssh/known_hosts`. > Where a fact is referenced in repo docs but not pinned in a config/script on this machine, @@ -21,7 +26,8 @@ | **ScadaBridge `infra`** | This Mac | Shared backing services (MSSQL, OPC-UA sims, SMTP, REST, Playwright) — **not** LDAP (see shared GLAuth below) | n/a (deps) | | **OtOpcUa `otopcua-dev`** | This Mac | 3 independent Akka clusters (MAIN + SITE-A + SITE-B) sharing one ConfigDb | `http://localhost:9200` | | **MxAccessGateway** | `windev` (10.100.0.48), Windows | Windows-native gRPC gateway + per-session x86 worker (no Docker) | `http://10.100.0.48:5120` (gRPC) | -| **Production (VD03)** | `wonder-app-vd03.zmr.zimmer.com` | Single-node ScadaBridge + MxGateway prod host | see `docs/operations/` runbooks | +| **DEV (VD03)** | `wonder-app-vd03.zmr.zimmer.com` | Single-node, all-in-one: ScadaBridge + OtOpcUa + MxGateway + HistorianGateway + Overview | see `docs/operations/` runbooks | +| **QA (VQ03/VQ04)** | `wonder-app-vq03` + `wonder-app-vq04` (+ `wonder-sql-vq03`) | **2-node HA pair** — ScadaBridge central+site pairs, OtOpcUa pair; singletons on VQ03 | `http://wonder-app-vq03.zmr.zimmer.com:5320` (Overview) | The three ScadaBridge stacks share one external Docker network **`scadabridge-net`**; the OtOpcUa `otopcua-dev` stack runs on its **own** default network (`otopcua-dev_default`) and is @@ -42,10 +48,21 @@ collide (see [§7](#7-consolidated-host-port-map)). | **This Mac** | local | macOS (darwin) | Dev workstation — runs all local Docker stacks | n/a | | **windev** | `10.100.0.48` | Windows | OtOpcUa Windows-service host **+** MxAccessGateway (gRPC `5120` / dashboard `5130`) | 22 | | **fixture host** | `10.100.0.35` | Debian/Linux + Docker | OtOpcUa driver **integration-test fixtures** + a test SQL Server | 22 | -| **VD03 (prod)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | Production single-node ScadaBridge + MxGateway | **2222** | +| **VD03 (DEV)** | `wonder-app-vd03.zmr.zimmer.com` | Windows | **DEV** all-in-one box: ScadaBridge (single-node) + OtOpcUa + MxGateway + HistorianGateway + Overview | **2222** | +| **VQ03 (QA)** | `wonder-app-vq03.zmr.zimmer.com` (`10.220.158.2`) | Windows | **QA node A** — ScadaBridge central-a + site-a, OtOpcUa node-a, **plus the singleton services**: MxGateway, HistorianGateway, Overview | **2222** | +| **VQ04 (QA)** | `wonder-app-vq04.zmr.zimmer.com` (`10.220.158.3`) | Windows | **QA node B** — ScadaBridge central-b + site-b, OtOpcUa node-b. **No** MxGateway/HistorianGateway/Overview | **2222** | +| **wonder-sql-vq03** | `wonder-sql-vq03.zmr.zimmer.com` | Windows | **QA data tier** — SQL Server (`1433`) + AVEVA Historian gRPC TLS (`32565`); mirrors `wonder-sql-vd03` | _(not probed)_ | | **gitea** | `gitea.dohertylan.com` (`10.100.0.228`) | Linux | Git remotes + NuGet feed (`/api/packages/dohertj2/nuget`) | 22 | -All are on the private `10.x` lab network — a LAN/VPN connection is required. +The lab hosts (`windev`, fixture, gitea) are on the private `10.x` network; the **wonder** hosts are +on the corporate `10.220.15x` network — both need the LAN/VPN. + +> ⚠️ **`vd` = DEV, `vq` = QA.** VD03 was labelled "production" throughout this doc until 2026-08-13; +> it is not. The convention is confirmed by the family's own MES spec (DEV `wonder-sql-vd01` vs QA +> `wondersqlqa`, [`mes-delmia-integration-api.md`](mes-delmia-integration-api.md) §1.1). Note VD03's +> `ASPNETCORE_ENVIRONMENT` genuinely **is** `Production` — that is a .NET config-overlay name and +> says nothing about the deployment tier; the two axes are orthogonal. **No production box has been +> identified** — do not assume either of these is it. ### 2.2 How to connect (passwordless SSH) @@ -67,9 +84,20 @@ Host windev |---|---|---| | **windev** (Win host) | `ssh windev` | Configured alias; user `dohertj2`, key `id_ed25519`, port 22 | | **fixture host** | `ssh dohertj2@10.100.0.35` | In `known_hosts`; **no** config alias — pass user explicitly; port 22, key-based | -| **VD03 (prod)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (the only non-standard SSH port); user/key not pinned in config — confirm before use | +| **VD03 (DEV)** | `ssh dohertj2@wonder-app-vd03.zmr.zimmer.com -p 2222` | In `known_hosts` on **port 2222** (servecli, not OpenSSH); user/key not pinned in config — confirm before use. Durable key `~/.ssh/servecli_wonder`. **servecli does not survive a reboot** — if it's down, use RDP | +| **VQ03 / VQ04 (QA)** | `ssh dohertj2@wonder-app-vq0{3,4}.zmr.zimmer.com -p 2222` | ⚠️ **Not working 2026-08-13** — see the banner note below | | **local Linux VMs** | `ssh orb` / `orb` | OrbStack-managed | +> ⚠️ **A TCP-open `2222` does NOT mean SSH works.** On 2026-08-13 all three wonder boxes +> (VD03, VQ03, VQ04) accepted a TCP connection on `2222` and then sent **no SSH banner** — the +> server must send its identification string first (RFC 4253 §4.2), so a bannerless socket is a +> listening port with no working SSH service behind it. The visible symptom is `ssh` hanging until +> it is killed, *not* a refused connection or an auth error. Every session attempt that day failed +> this way, so **no shell access to any wonder box was available** and nothing could be deployed. +> Probe it with a banner grab, never a port scan: +> `exec 3<>/dev/tcp//2222 && head -c 60 <&3`. +> servecli is known not to survive a reboot; the documented fallback is RDP. + > ⚠️ `~/bin` is **empty** on this Mac. OtOpcUa's `CLAUDE.md` mentions an `lmxopcua-fix` helper "in > `~/bin`" for controlling the `10.100.0.35` fixture containers — it is **not present here** (it's a > Windows-side helper). On this machine, drive the fixture host with direct SSH, e.g. @@ -253,8 +281,9 @@ dotnet build src/MxGateway.Worker/MxGateway.Worker.csproj -p:Platform=x86 - **Worker model:** the Server spawns one `ZB.MOM.WW.MxGateway.Worker.exe` (x86) **per gRPC session**; IPC over a named pipe (`\\.\pipe\mxgateway-` + a per-session `MXGATEWAY_WORKER_NONCE`); heartbeat 5s / grace 15s; max 64 concurrent sessions. The worker exits when the session closes. -- **Production hosts:** both `10.100.0.48` and `wonder-app-vd03` serve gRPC on `:5120` (per - `docs/GatewayConfiguration.md`). +- **Deployed hosts:** `10.100.0.48` (windev), `wonder-app-vd03` (DEV) and `wonder-app-vq03` (QA) + all serve gRPC on `:5120` (per `docs/GatewayConfiguration.md`; the VQ03 port was probed + 2026-08-13). **QA runs MxGateway on VQ03 only** — VQ04 has no gateway. ### 5.3 Who connects to it @@ -320,7 +349,58 @@ Every published host port across the local stacks (no collisions — all can run | 8089 | `otopcua-dev` traefik :8080 | Traefik dashboard | otopcua-dev | **Remote (non-local) endpoints:** MxAccessGateway gRPC `10.100.0.48:5120` (h2c) / dashboard `:5130`; -production gRPC on `wonder-app-vd03:5120`. SSH: windev/fixture/gitea on `22`, **VD03 on `2222`**. +DEV gRPC on `wonder-app-vd03:5120`, QA on `wonder-app-vq03:5120`. SSH: windev/fixture/gitea on `22`, +**VD03/VQ03/VQ04 on `2222`**. + +### 7.1 QA port map: VQ03 and VQ04 + +Established **2026-08-13** from two independent live sources, neither of them a config file — treat +connection strings and service names as unverified. Rows are marked with how each was established: + +- **`scan`** — TCP port scan of both hosts, plus a `/health/ready` GET where one was served. +- **`dash`** — read off the **QA Overview dashboard** (`vq03:5320`), which was already polling the + QA fleet and reports each instance's endpoint, health entries and active/standby state. Stronger + evidence than a scan for these rows: the dashboard is successfully talking to the service, not + merely finding an open socket. These ports were never scanned by me. + +| Port | VQ03 | VQ04 | How | Service | +|---|---|---|---|---| +| 9000 | ✅ | ✅ | dash | **OtOpcUa** host + AdminUI (same listener) | +| 4840 | ✅ | ✅ | scan | OtOpcUa **OPC UA** endpoint (`opc.tcp`) | +| 4053 | ✅ | ✅ | dash | OtOpcUa Akka remoting (from the reported leader address) | +| 8085 | ✅ | ✅ | scan | **ScadaBridge Central** — CentralUI + Management + Inbound API | +| 8084 | ✅ | ✅ | dash | ScadaBridge **Site** health listener (no UI; `/` → 404) | +| 8081 | ✅ | ✅ | scan | ScadaBridge Central Akka remoting | +| 8082 / 8083 | ✅ | ✅ | scan | ScadaBridge Site Akka / gRPC control plane (h2c — rejects an HTTP/1.1 GET) | +| 5120 / 5130 | ✅ | ✗ | scan | **MxAccessGateway** gRPC / dashboard | +| 5220 / 5221 | ✅ | ✗ | scan | **HistorianGateway** dashboard / gRPC h2c | +| 5320 | ✅ | ✗ | scan | **Overview** dashboard (the QA fleet's own single pane) | +| 1433 | ✅ | ✗ | scan | SQL Server | +| 2222 | ⚠️ | ⚠️ | scan | servecli SSH — **TCP-open but NOT serving SSH** (see below) | + +**Live state at capture** (`8 instances across 4 applications`, all Up): + +| Application | VQ03 | VQ04 | +|---|---|---| +| OtOpcUa (cluster `MAIN`) | node-a — **Standby** (`active 503`) | node-b — **Active** (`active 200`) | +| ScadaBridge `central` | central-a — **Active** | central-b — Standby | +| ScadaBridge `site` | site-a — **Active** | site-b — Standby | +| MxAccessGateway | Active | — | +| HistorianGateway | Up | — | + +Three things worth carrying forward: + +1. **QA is a real 2-node HA pair; DEV (VD03) is a single box.** So QA is the only environment here + where failover, leader election and the active/standby split are actually exercised. A change + that works on VD03 has not been tested against a pair. +2. **The active node is not the same for every application.** At capture, ScadaBridge's central and + site singletons were on VQ03 while OtOpcUa's active node was VQ04 — these are independent Akka + clusters and there is no reason for them to agree. Don't infer one from the other. +3. **QA's Overview registry can use real FQDNs, unlike VD03's.** `LeaderResolver` matches a reported + Akka leader to an instance **by host only**, which needs distinct hosts per instance — true on QA, + false on the co-located VD03 box, where the workaround is to point every `BaseUrl` at `127.0.0.1` + so resolution never matches and each chip falls back to the raw Akka address. On QA the leader + chips resolve to real names (`node-a (vq03)`), so that workaround is **not** wanted here. --- @@ -328,14 +408,35 @@ production gRPC on `wonder-app-vd03:5120`. SSH: windev/fixture/gitea on `22`, ** Every credential shown above (`OtOpcUa!Dev123`, `ScadaBridge_Dev1#`, the inline API-key peppers, the `docker-dev` JWT signing key, the `mxgw_…` API key) is a **dev-only placeholder** for the local -stacks — never reuse as a real secret. Production injects real secrets out-of-band (NSSM env / secret -store), per ScadaBridge `docs/operations/inbound-api-key-reissue.md` (the VD03 runbook). The full +stacks — never reuse as a real secret. The deployed wonder boxes (DEV + QA) inject real secrets +out-of-band (NSSM env / secret store), per ScadaBridge +`docs/operations/inbound-api-key-reissue.md`. Secrets are **per-stack** — a DEV value is never a QA +value, and inbound API keys in particular cannot be copied between them ([§9](#9-wonder-boxes-dev-and-qa)). The full 🔒 secret inventory and the `__`-env-var override forms are in [`env_vars.md`](env_vars.md) §5. -## 9. Production (VD03) — pointer +## 9. Wonder boxes: DEV and QA -`wonder-app-vd03.zmr.zimmer.com` (SSH `:2222`) runs the production single-node ScadaBridge and the -MxGateway (gRPC `:5120`). The production install is **not a scripted in-repo flow** here — the -operational procedures live in ScadaBridge `docs/operations/` (`failover-procedures.md`, -`maintenance-procedures.md`, `inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any -prod service/port specifics not in those runbooks as unverified. +`wonder-app-vd03.zmr.zimmer.com` (SSH `:2222`) is the **DEV** all-in-one box — single-node +ScadaBridge plus OtOpcUa, MxGateway (gRPC `:5120`), HistorianGateway and Overview. +`wonder-app-vq03` + `wonder-app-vq04` are the **QA** pair ([§7.1](#71-qa-port-map-vq03-and-vq04)). + +Neither install is a scripted in-repo flow — the operational procedures live in ScadaBridge +`docs/operations/` (`failover-procedures.md`, `maintenance-procedures.md`, +`inbound-api-key-reissue.md`, `troubleshooting-guide.md`). Treat any service/port specifics not in +those runbooks, or not probed in §7.1, as unverified. + +**Auth posture differs between the two, and QA is the correct one:** + +| | VD03 (DEV) | VQ03/VQ04 (QA) | +|---|---|---| +| `POST :8085/management` | **No auth** (`DisableLogin=true`) — `ListApiKeys`/`CreateApiKey` succeed unauthenticated from off-box | **Basic auth required** (`AUTH_FAILED` without it) | + +That is expected on a dev box and is **not** a finding there, but it means any DEV-only procedure +that relies on unauthenticated management calls **will not work on QA** — supply credentials. + +**Inbound API keys do not cross environments.** Each stack mints its own secrets (peppered-HMAC, +never recoverable) and config bundles deliberately carry no keys, so a QA `sbk_…` token 401s on DEV +and its keyId is absent from DEV's store entirely — verified 2026-08-13. Probe recipe: `POST +:8085/api/MesMoveIn -d '{}'` → **`400 VALIDATION_FAILED`** means the key is valid on that stack +(it reached the method and executed nothing); `401` means rejected. Pair it with a positive control +— auth runs *before* route resolution, so a bad **method name** also returns `401`.