# Infrastructure — testing & deployment

Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
depend on: the shared Docker host, the driver integration fixtures, the env-gated
**live-gate** recipes, the integration-test harness, the local dev rig, and the
gateway/deploy setups. This file is the single entry point; deeper history lives in
the linked docs.

> **Credentials below are dev-only** (per decision #137) — never production values.
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).

**Related deep docs**
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.

---

## 1. Shared Linux Docker host — `10.100.0.35`

All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
shared Debian host, **not** on the dev machine. Discover everything with the
`project=lmxopcua` label:

```bash
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
```

- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
  `docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
  Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
  where `/opt` needs sudo). Current layout:

  | Stack dir | Purpose |
  |---|---|
  | `/opt/otopcua-mssql` | central SQL (always-on) |
  | `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` · `/opt/otopcua-mqtt` · `/opt/otopcua-mtconnect` | driver fixtures |
  | `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
  | `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |

- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
  deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.

## 2. Central SQL Server (always-on)

| | |
|---|---|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
| Endpoint | `10.100.0.35:14330` → `1433` |
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |

Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.

## 3. Driver integration fixtures

Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
`tests/Drivers/…​.IntegrationTests/Docker/`.

| Driver | Fixture image | Endpoint | Bring up (on the host) |
|---|---|---|---|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
| MQTT | `eclipse-mosquitto:2.0.22` (+ a publisher sidecar) | `10.100.0.35:8883` TLS+auth · `:1883` plaintext+auth | `MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d` (stack `/opt/otopcua-mqtt`) |
| MQTT — Sparkplug B | `otopcua-sparkplug-sim` (project-owned C# edge-node simulator) | same broker; group **`OtOpcUaSim`**, edge nodes **`EdgeA`** / **`EdgeB`**, `EdgeA` device **`Filler1`**, ~2 s cadence | `docker compose --profile sparkplug up -d` (same stack). Answers rebirth NCMDs; restart it to force a fresh birth |
| MTConnect | `mtconnect/agent:2.7.0.12` + `python:3.13-alpine` adapter | `http://10.100.0.35:5000` | `docker compose up -d --wait` |

**Endpoint overrides** point a suite at a real PLC instead of the sim:
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT` ·
`MQTT_FIXTURE_ENDPOINT` / `MQTT_FIXTURE_PLAIN_ENDPOINT` · `MTCONNECT_AGENT_ENDPOINT`.

> **MQTT needs credentials + material, unlike the other fixtures.** The broker has **no anonymous
> fallback on either listener** (deliberate — an anonymous broker would let a bad auth config pass),
> so it will not even start until `./gen-fixture-material.sh` has written `secrets/` (password file +
> CA + server cert/key, gitignored) and `MQTT_FIXTURE_USERNAME`/`MQTT_FIXTURE_PASSWORD` are exported
> at `docker compose up`. The live suite additionally wants `MQTT_FIXTURE_CA_CERT` pointing at a local
> copy of the CA: `scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt`.
> It is also the only fixture whose services carry the `project: lmxopcua` label (see CLAUDE.md).

> **MTConnect is a TWO-service stack.** The `mtconnect/agent` image ships no simulator, so a lone
> Agent answers `/probe` and reports every observation `UNAVAILABLE` forever. The `adapter` service
> (a stdlib SHDR feeder, `Docker/adapter.py`) is the data source; the Agent dials out to it. Its
> suite also probes over **HTTP**, not TCP — an Agent's port accepts connections before its device
> model is parsed. Full detail + the seeded device model:
> [`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md`](../tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md).

> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
> may leave a stale container bound. Force-remove before switching profiles:
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.

## 4. Host.IntegrationTests real-mode harness

`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).

| Dep | Container | Endpoint | Notes |
|---|---|---|---|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |

Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).

```bash
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
export OTOPCUA_HARNESS_USE_SQL=1  OTOPCUA_HARNESS_USE_LDAP=1
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35        # harness uses port 3894
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
```

## 5. Live-gate recipes (env-gated destructive tests)

These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
gated by env vars and always restores the shared resource in a `finally`.

### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.

```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
```

### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
Drives the directory-outage circuit through the production auth path against the harness GLAuth
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.

```bash
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
export GLAUTH_LIVE_HOST=10.100.0.35   # port defaults to 3894 (GLAUTH_LIVE_PORT)
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
```

### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.

```bash
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221   # local gw, h2c
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true              # required for h2c
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder   # gw ApiKeys:Mode=Disabled
export HISTGW_TEST_TAG=SysTimeSec
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
```

## 6. HistorianGateway — deploy for live testing

The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.

- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
  (`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
  dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
  `docker/.env` / secrets, never in committed compose.
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
  in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.

## 7. docker-dev local rig — `docker-dev/docker-compose.yml`

A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.

| Surface | Endpoint | Notes |
|---|---|---|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |

```bash
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
# headless deploy: POST http://localhost:9200/api/deployments  (X-Api-Key)
docker compose -f docker-dev/docker-compose.yml down -v   # reset
```

## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`

The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).

## 9. Deployment (production rig)

The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
and a real directory (LDAP) — never the dev values in this doc.

> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
