Joseph Doherty
07a4a5ff5e
Task 14 — the P1 milestone gate. Ran the live /run verification on a docker-dev
rig against the real Mosquitto TLS+auth fixture, and recorded the result.
The gate did its job: it found MQTT was unauthorable in production.
RawDriverTypeDialog's driver-type list is a hand-maintained array that nobody
added MQTT to, so the /raw "New driver" picker never offered it — with every
other layer (contracts, driver, browser, factory, host registration, typed tag
editor) complete and 266 green unit tests. Nothing tied that array to
DriverTypeNames and AdminUI has no bUnit, so no offline test could see it.
Fixed, plus RawDriverTypeDialogParityTests: a reflection parity guard that
fails for ANY DriverTypeNames entry missing from the picker. Verified
falsifiable — RED with "…cannot be authored from the /raw New-driver picker,
so they are unreachable in production: Mqtt" before the one-line fix.
A second gap is left OPEN and documented, not fixed (no task ever covered it):
MqttDriverForm / MqttDeviceForm do not exist, and DriverConfigModal/DeviceModal
have no raw-JSON fallback — so an operator can create an MQTT driver but cannot
author its broker endpoint or credentials. P1 is not operator-complete until
those two razor forms land. The gate authored config via SQL to get past it.
Live gate result (isolated 2-node MAIN stack, image built from this branch,
fixture at 10.100.0.35:8883, AllowUntrustedServerCertificate rather than
pinning the CA into the rig):
- typed MQTT tag editor renders and round-trips; Json shows the JSON-path
field, Raw/Scalar hide it; wildcard topic a/+/c blocked at save; data-type
list offers Float64 and no Double; qos/retainSeed absent on "(driver
default)" and qos:2 written as a number; isHistorized/historianTagname
survive a topic-only edit; SparkplugB renders the P2 placeholder and
switching back keeps Plain values; jsonPath is guidance not a gate in all
three shapes ($ seeded only for a brand-new blank tag, blank stays absent,
clearing saves fine)
- deployed, and both editor-written blobs resolved and served changing live
values through OPC UA (22.8 / 1629, StatusCode Good) — no BadNodeIdUnknown
- the bespoke #-observation browser drove a real broker session and rendered
the fixture's actual topic tree (line1/{counter,state,temperature},
noise/chatter, retained/seed); a Modbus device's picker still correctly
reports "Browsing unavailable"
Also found live and documented (not a code change): both nodes of a redundant
pair run the driver, so a FIXED clientId makes them evict each other forever —
the broker logs "already connected, closing old connection" and both nodes
reconnect every ~2s while still reporting Healthy. Unset (the default) is
correct; confirmed 0 reconnects/60s on both nodes after removing it.
Suites: offline 1134 passed / 0 failed (266 Driver.Mqtt + 730 AdminUI incl. the
3 new guards + 138 Core.Abstractions); live 7/7 against the fixture.
Docs: new docs/drivers/Mqtt.md + Mqtt-Test-Fixture.md (the per-driver + fixture
convention every other driver follows), MQTT rows in docs/drivers/README.md,
TestConnectProbes.md, root README.md, and infra/README.md §3.
CLAUDE.md drift corrected while adding the MQTT endpoints:
- the "every fixture carries project: lmxopcua" claim was false — only the
MQTT fixture does; the filter returns one stack, not the fleet
- lmxopcua-fix.ps1 is Windows-VM-only; on macOS drive the host over ssh/rsync
(and the docker-dev rig itself runs locally under OrbStack)
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
204 lines
12 KiB
Markdown
204 lines
12 KiB
Markdown
# Infrastructure — testing & deployment
|
|
|
|
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
|
|
depend on: the shared Docker host, the driver integration fixtures, the env-gated
|
|
**live-gate** recipes, the integration-test harness, the local dev rig, and the
|
|
gateway/deploy setups. This file is the single entry point; deeper history lives in
|
|
the linked docs.
|
|
|
|
> **Credentials below are dev-only** (per decision #137) — never production values.
|
|
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
|
|
|
|
**Related deep docs**
|
|
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
|
|
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
|
|
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
|
|
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
|
|
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
|
|
|
|
---
|
|
|
|
## 1. Shared Linux Docker host — `10.100.0.35`
|
|
|
|
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
|
|
shared Debian host, **not** on the dev machine. Discover everything with the
|
|
`project=lmxopcua` label:
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
|
|
```
|
|
|
|
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
|
|
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
|
|
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
|
|
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
|
|
where `/opt` needs sudo). Current layout:
|
|
|
|
| Stack dir | Purpose |
|
|
|---|---|
|
|
| `/opt/otopcua-mssql` | central SQL (always-on) |
|
|
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` · `/opt/otopcua-mqtt` | driver fixtures |
|
|
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
|
|
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
|
|
|
|
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
|
|
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
|
|
|
|
## 2. Central SQL Server (always-on)
|
|
|
|
| | |
|
|
|---|---|
|
|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
|
|
| Endpoint | `10.100.0.35:14330` → `1433` |
|
|
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
|
|
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
|
|
|
|
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
|
|
|
|
## 3. Driver integration fixtures
|
|
|
|
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
|
|
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
|
|
`tests/Drivers/….IntegrationTests/Docker/`.
|
|
|
|
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|
|
|---|---|---|---|
|
|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
|
|
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
|
|
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
|
|
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
|
|
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
|
|
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
|
|
| MQTT | `eclipse-mosquitto:2.0.22` (+ a publisher sidecar) | `10.100.0.35:8883` TLS+auth · `:1883` plaintext+auth | `MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d` (stack `/opt/otopcua-mqtt`) |
|
|
|
|
**Endpoint overrides** point a suite at a real PLC instead of the sim:
|
|
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT` ·
|
|
`MQTT_FIXTURE_ENDPOINT` / `MQTT_FIXTURE_PLAIN_ENDPOINT`.
|
|
|
|
> **MQTT needs credentials + material, unlike the other fixtures.** The broker has **no anonymous
|
|
> fallback on either listener** (deliberate — an anonymous broker would let a bad auth config pass),
|
|
> so it will not even start until `./gen-fixture-material.sh` has written `secrets/` (password file +
|
|
> CA + server cert/key, gitignored) and `MQTT_FIXTURE_USERNAME`/`MQTT_FIXTURE_PASSWORD` are exported
|
|
> at `docker compose up`. The live suite additionally wants `MQTT_FIXTURE_CA_CERT` pointing at a local
|
|
> copy of the CA: `scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt`.
|
|
> It is also the only fixture whose services carry the `project: lmxopcua` label (see CLAUDE.md).
|
|
|
|
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
|
|
> may leave a stale container bound. Force-remove before switching profiles:
|
|
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
|
|
|
|
## 4. Host.IntegrationTests real-mode harness
|
|
|
|
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
|
|
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
|
|
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
|
|
|
|
| Dep | Container | Endpoint | Notes |
|
|
|---|---|---|---|
|
|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
|
|
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
|
|
|
|
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
|
|
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
|
|
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
|
|
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
|
|
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
|
|
```
|
|
|
|
## 5. Live-gate recipes (env-gated destructive tests)
|
|
|
|
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
|
|
gated by env vars and always restores the shared resource in a `finally`.
|
|
|
|
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
|
|
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
|
|
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
|
|
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
|
|
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
|
|
```
|
|
|
|
### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
|
|
Drives the directory-outage circuit through the production auth path against the harness GLAuth
|
|
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
|
|
export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT)
|
|
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
|
|
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
|
|
```
|
|
|
|
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
|
|
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
|
|
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
|
|
|
|
```bash
|
|
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
|
|
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
|
|
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
|
|
export HISTGW_TEST_TAG=SysTimeSec
|
|
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
|
|
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
|
|
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
|
|
```
|
|
|
|
## 6. HistorianGateway — deploy for live testing
|
|
|
|
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
|
|
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
|
|
|
|
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
|
|
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
|
|
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
|
|
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
|
|
`docker/.env` / secrets, never in committed compose.
|
|
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
|
|
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
|
|
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
|
|
|
|
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
|
|
|
|
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
|
|
|
|
| Surface | Endpoint | Notes |
|
|
|---|---|---|
|
|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
|
|
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
|
|
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
|
|
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
|
|
|
|
```bash
|
|
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
|
|
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
|
|
docker compose -f docker-dev/docker-compose.yml down -v # reset
|
|
```
|
|
|
|
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
|
|
|
|
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
|
|
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
|
|
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
|
|
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
|
|
|
|
## 9. Deployment (production rig)
|
|
|
|
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
|
|
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
|
|
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
|
|
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
|
|
and a real directory (LDAP) — never the dev values in this doc.
|
|
|
|
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
|
|
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
|