diff --git a/CLAUDE.md b/CLAUDE.md index 07c01e17..309e8d26 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -125,6 +125,11 @@ central SQL Server. ## Docker Workflow (driver fixtures + central SQL Server) +> **Infra index:** [`infra/README.md`](infra/README.md) is the canonical index of all testing + +> deployment infrastructure — the Docker host layout, driver fixtures, the env-gated **live-gate** +> recipes (S7 blackhole, GLAuth outage, HistorianGateway LiveIntegration), the integration-test +> harness, the docker-dev rig, and the deploy setup. Start there; the sections below + `docs/v2/dev-environment.md` carry the detail. + > **Migrated 2026-04-28**: Docker config + host moved off this dev VM (DESKTOP-6JL3KKO) onto the shared Linux Docker host (`DOCKER`, 10.100.0.35) so the dev VM could shed WSL2/Hyper-V and have its GPU re-attached via ESXi passthrough. Docker Desktop is no longer installed here. All checked-in `appsettings.json` defaults, fixture-class default endpoints, and `e2e-config.sample.json` were rewritten to target `10.100.0.35`. The driver fixture compose files under `tests/.../Docker/docker-compose.yml` now carry a `project: lmxopcua` label on every service. See `docs/v2/dev-environment.md` for the full rewrite (header dated 2026-04-28). Docker workloads run on a shared Linux host at **`10.100.0.35`** — not on this VM. Stacks live at `/opt/otopcua-/` on the host and carry the `project=lmxopcua` label so they're discoverable via `docker ps --filter label=project=lmxopcua`. diff --git a/infra/README.md b/infra/README.md new file mode 100644 index 00000000..06c69bb4 --- /dev/null +++ b/infra/README.md @@ -0,0 +1,193 @@ +# Infrastructure — testing & deployment + +Canonical index of every piece of infrastructure OtOpcUa's tests and deployments +depend on: the shared Docker host, the driver integration fixtures, the env-gated +**live-gate** recipes, the integration-test harness, the local dev rig, and the +gateway/deploy setups. This file is the single entry point; deeper history lives in +the linked docs. + +> **Credentials below are dev-only** (per decision #137) — never production values. +> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP). + +**Related deep docs** +- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap. +- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook. +- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps. +- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook. +- [`reference_histgw_local_deploy`], [`reference_ww_historian_vm`] (memory) — HistorianGateway deploy detail. + +--- + +## 1. Shared Linux Docker host — `10.100.0.35` + +All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a +shared Debian host, **not** on the dev machine. Discover everything with the +`project=lmxopcua` label: + +```bash +ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"' +``` + +- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`). + `docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the + Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly. +- **Stacks** live at `/opt/otopcua-/` (root-owned) or `~/otopcua-/` (user-owned, + where `/opt` needs sudo). Current layout: + + | Stack dir | Purpose | + |---|---| + | `/opt/otopcua-mssql` | central SQL (always-on) | + | `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` | driver fixtures | + | `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) | + | `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 | + +- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored + deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`. + +## 2. Central SQL Server (always-on) + +| | | +|---|---| +| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` | +| Endpoint | `10.100.0.35:14330` → `1433` | +| Creds (dev) | `sa` / `OtOpcUaDev_2026!` | +| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) | + +Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`. + +## 3. Driver integration fixtures + +Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly** +when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under +`tests/Drivers/…​.IntegrationTests/Docker/`. + +| Driver | Fixture image | Endpoint | Bring up (on the host) | +|---|---|---|---| +| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile up -d` | +| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile up -d` | +| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) | +| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` | +| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` | +| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) | + +**Endpoint overrides** point a suite at a real PLC instead of the sim: +`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT`. + +> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down` +> may leave a stale container bound. Force-remove before switching profiles: +> `docker rm -f $(docker ps -aq --filter name=otopcua-)` then `up --force-recreate`. + +## 4. Host.IntegrationTests real-mode harness + +`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by +default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack: +`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`). + +| Dep | Container | Endpoint | Notes | +|---|---|---|---| +| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 | +| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below | + +Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`): +`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only). + +```bash +ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d' +export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1 +export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331 +export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894 +dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests +ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down' +``` + +## 5. Live-gate recipes (env-gated destructive tests) + +These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is +gated by env vars and always restores the shared resource in a `finally`. + +### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`) +Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart` +can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN. + +```bash +ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' +export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"' +export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"' +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage" +``` + +### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`) +Drives the directory-outage circuit through the production auth path against the harness GLAuth +(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly. + +```bash +ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap' +export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT) +export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"' +export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"' +dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive" +``` + +### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN +Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires +the corporate **VPN up** and a running gateway (see §6). Full deploy detail: [`reference_histgw_local_deploy`]. + +```bash +export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c +export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c +export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled +export HISTGW_TEST_TAG=SysTimeSec +export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox +export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox +export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation +dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration" +``` + +## 6. HistorianGateway — deploy for live testing + +The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the +**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths. + +- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433` + (`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`. +- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`, + dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored + `docker/.env` / secrets, never in committed compose. +- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp` + in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on. +- Full recipe + gotchas: [`reference_histgw_local_deploy`] · historian VM: [`reference_ww_historian_vm`]. + +## 7. docker-dev local rig — `docker-dev/docker-compose.yml` + +A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification. + +| Surface | Endpoint | Notes | +|---|---|---| +| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change | +| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane | +| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed | +| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only | + +```bash +docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik +# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key) +docker compose -f docker-dev/docker-compose.yml down -v # reset +``` + +## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893` + +The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness +GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind +`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of +truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/). + +## 9. Deployment (production rig) + +The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as +NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with +`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes → +restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL) +and a real directory (LDAP) — never the dev values in this doc. + +> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the +> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).