# Infrastructure — testing & deployment Canonical index of every piece of infrastructure OtOpcUa's tests and deployments depend on: the shared Docker host, the driver integration fixtures, the env-gated **live-gate** recipes, the integration-test harness, the local dev rig, and the gateway/deploy setups. This file is the single entry point; deeper history lives in the linked docs. > **Credentials below are dev-only** (per decision #137) — never production values. > Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP). **Related deep docs** - [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap. - [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook. - `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps. - [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook. - `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail. --- ## 1. Shared Linux Docker host — `10.100.0.35` All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a shared Debian host, **not** on the dev machine. Discover everything with the `project=lmxopcua` label: ```bash ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"' ``` - **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`). `docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly. - **Stacks** live at `/opt/otopcua-/` (root-owned) or `~/otopcua-/` (user-owned, where `/opt` needs sudo). Current layout: | Stack dir | Purpose | |---|---| | `/opt/otopcua-mssql` | central SQL (always-on) | | `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` | driver fixtures | | `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) | | `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 | - The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`. ## 2. Central SQL Server (always-on) | | | |---|---| | Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` | | Endpoint | `10.100.0.35:14330` → `1433` | | Creds (dev) | `sa` / `OtOpcUaDev_2026!` | | Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) | Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`. ## 3. Driver integration fixtures Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly** when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under `tests/Drivers/…​.IntegrationTests/Docker/`. | Driver | Fixture image | Endpoint | Bring up (on the host) | |---|---|---|---| | Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile up -d` | | AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile up -d` | | AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) | | S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` | | OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` | | FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) | **Endpoint overrides** point a suite at a real PLC instead of the sim: `MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT`. > **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down` > may leave a stale container bound. Force-remove before switching profiles: > `docker rm -f $(docker ps -aq --filter name=otopcua-)` then `up --force-recreate`. ## 4. Host.IntegrationTests real-mode harness `tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack: `~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`). | Dep | Container | Endpoint | Notes | |---|---|---|---| | SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 | | GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below | Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`): `serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only). ```bash ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d' export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1 export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331 export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894 dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down' ``` ## 5. Live-gate recipes (env-gated destructive tests) These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is gated by env vars and always restores the shared resource in a `finally`. ### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`) Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart` can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN. ```bash ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d' export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"' export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"' dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage" ``` ### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`) Drives the directory-outage circuit through the production auth path against the harness GLAuth (§4), paused/unpaused. Also verifies the GLAuth swap binds correctly. ```bash ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap' export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT) export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"' export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"' dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive" ``` ### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`. ```bash export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled export HISTGW_TEST_TAG=SysTimeSec export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration" ``` ## 6. HistorianGateway — deploy for live testing The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the **real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths. - **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433` (`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`. - **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`, dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored `docker/.env` / secrets, never in committed compose. - **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp` in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on. - Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`. ## 7. docker-dev local rig — `docker-dev/docker-compose.yml` A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification. | Surface | Endpoint | Notes | |---|---|---| | AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change | | OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane | | Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed | | LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only | ```bash docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik # headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key) docker compose -f docker-dev/docker-compose.yml down -v # reset ``` ## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893` The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind `cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/). ## 9. Deployment (production rig) The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with `scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes → restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL) and a real directory (LDAP) — never the dev values in this doc. > The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the > `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).