Joseph Doherty
cf9ce91e51
The first and only thing in the MTConnect workstream that exercises the driver
against a real Agent; everything else runs on canned XML.
Fixture (Docker/): two services, both stock images with this folder bind-mounted.
- `agent` mtconnect/agent:2.7.0.12 on :5000. NOTE the repo name: the source
project is mtconnect/cppagent but the PUBLISHED image is
mtconnect/agent; mtconnect/cppagent does not exist on Docker Hub.
- `adapter` a stdlib SHDR feeder. Required, not optional: the agent image ships
only the binary + schemas/styles, so a lone Agent answers /probe and
then reports every observation UNAVAILABLE forever, which can prove
nothing about reads or streaming.
Devices.xml seeds one DataItem per inference branch (SAMPLE+units, TIME_SERIES,
PART_COUNT/LINE_NUMBER, controlled vocab, DATA_SET, CONDITION), gives every named
item `name != id` (the inverse of the hand-authored unit fixtures), and leaves one
EVENT permanently unfed so UNAVAILABLE -> BadNoCommunication has a live subject.
Suite (12 tests) asserts STRUCTURALLY against the Agent's own /probe response, never
by literal id, so it survives an edit to Devices.xml and can be pointed at a real
machine tool via MTCONNECT_AGENT_ENDPOINT. It skips cleanly (12/12) when no Agent
answers, and each test carries a hard [Fact(Timeout)] so a half-up fixture cannot
wedge a build.
Three findings from bringing the fixture up, each now pinned in a comment:
- agent.cfg must be pure ASCII; one non-ASCII byte in a COMMENT makes the config
parser reject the whole file with a bare "Failed / Stopped at line: N" and exit.
- `sampleCount` is an attribute of a TIME_SERIES observation, not of a DataItem
declaration. A real Agent meeting one on a declaration DROPS THE ENTIRE DATA ITEM
from the device model, so the inference only ever sees null and a live TIME_SERIES
tag is always variable-length. Asserted, not ignored.
- The Agent's own <Agent> self-model publishes update-rate SAMPLEs that tick whether
or not any adapter is attached. Excluding them is load-bearing: with them included
the "live stream delivered a changed value" test passed with the fixture's data
source deliberately stopped.
MTConnectError-under-HTTP-200 is NOT covered live and cannot be: cppagent 2.7 answers
an unknown device 404 and an out-of-range sequence 400, each with a well-formed error
body. That shape stays canned-only, and the suite says so.
213 lines
13 KiB
Markdown
213 lines
13 KiB
Markdown
# Infrastructure — testing & deployment
|
|
|
|
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
|
|
depend on: the shared Docker host, the driver integration fixtures, the env-gated
|
|
**live-gate** recipes, the integration-test harness, the local dev rig, and the
|
|
gateway/deploy setups. This file is the single entry point; deeper history lives in
|
|
the linked docs.
|
|
|
|
> **Credentials below are dev-only** (per decision #137) — never production values.
|
|
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
|
|
|
|
**Related deep docs**
|
|
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
|
|
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
|
|
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
|
|
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
|
|
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
|
|
|
|
---
|
|
|
|
## 1. Shared Linux Docker host — `10.100.0.35`
|
|
|
|
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
|
|
shared Debian host, **not** on the dev machine. Discover everything with the
|
|
`project=lmxopcua` label:
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
|
|
```
|
|
|
|
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
|
|
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
|
|
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
|
|
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
|
|
where `/opt` needs sudo). Current layout:
|
|
|
|
| Stack dir | Purpose |
|
|
|---|---|
|
|
| `/opt/otopcua-mssql` | central SQL (always-on) |
|
|
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` · `/opt/otopcua-mqtt` · `/opt/otopcua-mtconnect` | driver fixtures |
|
|
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
|
|
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
|
|
|
|
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
|
|
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
|
|
|
|
## 2. Central SQL Server (always-on)
|
|
|
|
| | |
|
|
|---|---|
|
|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
|
|
| Endpoint | `10.100.0.35:14330` → `1433` |
|
|
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
|
|
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
|
|
|
|
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
|
|
|
|
## 3. Driver integration fixtures
|
|
|
|
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
|
|
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
|
|
`tests/Drivers/….IntegrationTests/Docker/`.
|
|
|
|
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|
|
|---|---|---|---|
|
|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
|
|
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
|
|
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
|
|
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
|
|
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
|
|
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
|
|
| MQTT | `eclipse-mosquitto:2.0.22` (+ a publisher sidecar) | `10.100.0.35:8883` TLS+auth · `:1883` plaintext+auth | `MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d` (stack `/opt/otopcua-mqtt`) |
|
|
| MQTT — Sparkplug B | `otopcua-sparkplug-sim` (project-owned C# edge-node simulator) | same broker; group **`OtOpcUaSim`**, edge nodes **`EdgeA`** / **`EdgeB`**, `EdgeA` device **`Filler1`**, ~2 s cadence | `docker compose --profile sparkplug up -d` (same stack). Answers rebirth NCMDs; restart it to force a fresh birth |
|
|
| MTConnect | `mtconnect/agent:2.7.0.12` + `python:3.13-alpine` adapter | `http://10.100.0.35:5000` | `docker compose up -d --wait` |
|
|
|
|
**Endpoint overrides** point a suite at a real PLC instead of the sim:
|
|
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT` ·
|
|
`MQTT_FIXTURE_ENDPOINT` / `MQTT_FIXTURE_PLAIN_ENDPOINT` · `MTCONNECT_AGENT_ENDPOINT`.
|
|
|
|
> **MQTT needs credentials + material, unlike the other fixtures.** The broker has **no anonymous
|
|
> fallback on either listener** (deliberate — an anonymous broker would let a bad auth config pass),
|
|
> so it will not even start until `./gen-fixture-material.sh` has written `secrets/` (password file +
|
|
> CA + server cert/key, gitignored) and `MQTT_FIXTURE_USERNAME`/`MQTT_FIXTURE_PASSWORD` are exported
|
|
> at `docker compose up`. The live suite additionally wants `MQTT_FIXTURE_CA_CERT` pointing at a local
|
|
> copy of the CA: `scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt`.
|
|
> It is also the only fixture whose services carry the `project: lmxopcua` label (see CLAUDE.md).
|
|
|
|
> **MTConnect is a TWO-service stack.** The `mtconnect/agent` image ships no simulator, so a lone
|
|
> Agent answers `/probe` and reports every observation `UNAVAILABLE` forever. The `adapter` service
|
|
> (a stdlib SHDR feeder, `Docker/adapter.py`) is the data source; the Agent dials out to it. Its
|
|
> suite also probes over **HTTP**, not TCP — an Agent's port accepts connections before its device
|
|
> model is parsed. Full detail + the seeded device model:
|
|
> [`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md`](../tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md).
|
|
|
|
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
|
|
> may leave a stale container bound. Force-remove before switching profiles:
|
|
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
|
|
|
|
## 4. Host.IntegrationTests real-mode harness
|
|
|
|
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
|
|
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
|
|
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
|
|
|
|
| Dep | Container | Endpoint | Notes |
|
|
|---|---|---|---|
|
|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
|
|
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
|
|
|
|
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
|
|
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
|
|
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
|
|
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
|
|
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
|
|
```
|
|
|
|
## 5. Live-gate recipes (env-gated destructive tests)
|
|
|
|
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
|
|
gated by env vars and always restores the shared resource in a `finally`.
|
|
|
|
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
|
|
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
|
|
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
|
|
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
|
|
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
|
|
```
|
|
|
|
### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
|
|
Drives the directory-outage circuit through the production auth path against the harness GLAuth
|
|
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
|
|
export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT)
|
|
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
|
|
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
|
|
```
|
|
|
|
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
|
|
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
|
|
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
|
|
|
|
```bash
|
|
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
|
|
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
|
|
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
|
|
export HISTGW_TEST_TAG=SysTimeSec
|
|
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
|
|
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
|
|
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
|
|
```
|
|
|
|
## 6. HistorianGateway — deploy for live testing
|
|
|
|
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
|
|
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
|
|
|
|
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
|
|
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
|
|
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
|
|
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
|
|
`docker/.env` / secrets, never in committed compose.
|
|
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
|
|
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
|
|
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
|
|
|
|
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
|
|
|
|
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
|
|
|
|
| Surface | Endpoint | Notes |
|
|
|---|---|---|
|
|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
|
|
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
|
|
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
|
|
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
|
|
|
|
```bash
|
|
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
|
|
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
|
|
docker compose -f docker-dev/docker-compose.yml down -v # reset
|
|
```
|
|
|
|
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
|
|
|
|
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
|
|
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
|
|
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
|
|
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
|
|
|
|
## 9. Deployment (production rig)
|
|
|
|
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
|
|
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
|
|
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
|
|
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
|
|
and a real directory (LDAP) — never the dev values in this doc.
|
|
|
|
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
|
|
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
|