dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fcbfb5618147db1a014a548f84dfd1b887b85cc2
lmxopcua/infra/README.md
T
Joseph Doherty 65fda7237f refactor(acl): delete the LDAP data-plane authenticator and its tests (#520) 
Stage 2b. LdapOpcUaUserAuthenticator was left registered nowhere by stage 2a;
this removes it, its three test files, and the config surface that existed only
to serve it.

Deleted: LdapOpcUaUserAuthenticator, LdapOpcUaUserAuthenticatorTests,
LdapAuthResilienceTests, LdapGlauthOutageLiveTests (19 tests).

Also removed four Security:Ldap keys — AuthTimeoutMs,
MaxConcurrentAuthentications, OutageFailureThreshold, OutageCooldownSeconds —
plus their validator rules and binding/validator test coverage. Verified first
that the ONLY consumers were the deleted class and the validator rules asserting
them; none appear in any checked-in appsettings. Leaving them would have created
exactly the dead-config surface #521/#524/#525 were filed to clear up.

Worth stating why that resilience machinery does not need relocating rather than
just deleting. It existed because the OPC UA SDK raises impersonation
SYNCHRONOUSLY under a SessionManager-wide event lock, so a hung LDAP bind there
serialized every session activation server-wide — Anonymous and X509 included.
Key verification is one indexed lookup plus one HMAC with no network I/O, so the
hazard the wall-clock bound, concurrency cap and outage circuit guarded against
no longer exists on that path. LDAP now runs only on AdminUI login, an ordinary
async request path where a slow directory affects the one request making it.

ConnectionTimeoutMs is RETAINED: it is projected into the shared library and is
not authenticator-specific.

Docs corrected rather than left to rot:
- CLAUDE.md's "LDAP Authentication" section described the deleted file as the
  data-plane authenticator. Rewritten, including a warning not to "restore" it on
  seeing the LDAP wiring that AdminUI legitimately still needs.
- infra/README §5b (the GLAuth outage live gate) marked RETIRED with the reason,
  rather than silently dropped — the harness GLAuth is still used by
  OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected.
- docs/security.md's five-key resilience table cut to the one surviving key, and
  a new API-key identity section added.
Historical archreview/ records are left as-is; they are dated accounts.

Host.IntegrationTests 211 tests (was 230, i.e. exactly the 19 removed) with the
same 3 pre-existing failures verified against clean master. Host 37, Security 89.

Claude-Session: https://claude.ai/code/session_015p7wGqy3YpZNCpDzTpGMKo
2026-07-31 13:08:50 -04:00

212 lines
13 KiB
Markdown
Raw Blame History

# Infrastructure — testing & deployment
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
depend on: the shared Docker host, the driver integration fixtures, the env-gated
**live-gate** recipes, the integration-test harness, the local dev rig, and the
gateway/deploy setups. This file is the single entry point; deeper history lives in
the linked docs.
> **Credentials below are dev-only** (per decision #137) — never production values.
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
**Related deep docs**
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
---
## 1. Shared Linux Docker host — `10.100.0.35`
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
shared Debian host, **not** on the dev machine. Discover everything with the
`project=lmxopcua` label:
```bash
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
```
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
where `/opt` needs sudo). Current layout:
| Stack dir | Purpose |
|---|---|
| `/opt/otopcua-mssql` | central SQL (always-on) |
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` · `/opt/otopcua-mqtt` · `/opt/otopcua-mtconnect` | driver fixtures |
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
## 2. Central SQL Server (always-on)
| | |
|---|---|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
| Endpoint | `10.100.0.35:14330` → `1433` |
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
## 3. Driver integration fixtures
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
`tests/Drivers/…​.IntegrationTests/Docker/`.
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|---|---|---|---|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
| MQTT | `eclipse-mosquitto:2.0.22` (+ a publisher sidecar) | `10.100.0.35:8883` TLS+auth · `:1883` plaintext+auth | `MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d` (stack `/opt/otopcua-mqtt`) |
| MQTT — Sparkplug B | `otopcua-sparkplug-sim` (project-owned C# edge-node simulator) | same broker; group **`OtOpcUaSim`**, edge nodes **`EdgeA`** / **`EdgeB`**, `EdgeA` device **`Filler1`**, ~2 s cadence | `docker compose --profile sparkplug up -d` (same stack). Answers rebirth NCMDs; restart it to force a fresh birth |
| MTConnect | `mtconnect/agent:2.7.0.12` + `python:3.13-alpine` adapter | `http://10.100.0.35:5000` | `docker compose up -d --wait` |
**Endpoint overrides** point a suite at a real PLC instead of the sim:
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT` ·
`MQTT_FIXTURE_ENDPOINT` / `MQTT_FIXTURE_PLAIN_ENDPOINT` · `MTCONNECT_AGENT_ENDPOINT`.
> **MQTT needs credentials + material, unlike the other fixtures.** The broker has **no anonymous
> fallback on either listener** (deliberate — an anonymous broker would let a bad auth config pass),
> so it will not even start until `./gen-fixture-material.sh` has written `secrets/` (password file +
> CA + server cert/key, gitignored) and `MQTT_FIXTURE_USERNAME`/`MQTT_FIXTURE_PASSWORD` are exported
> at `docker compose up`. The live suite additionally wants `MQTT_FIXTURE_CA_CERT` pointing at a local
> copy of the CA: `scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt`.
> It is also the only fixture whose services carry the `project: lmxopcua` label (see CLAUDE.md).
> **MTConnect is a TWO-service stack.** The `mtconnect/agent` image ships no simulator, so a lone
> Agent answers `/probe` and reports every observation `UNAVAILABLE` forever. The `adapter` service
> (a stdlib SHDR feeder, `Docker/adapter.py`) is the data source; the Agent dials out to it. Its
> suite also probes over **HTTP**, not TCP — an Agent's port accepts connections before its device
> model is parsed. Full detail + the seeded device model:
> [`tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md`](../tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md).
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
> may leave a stale container bound. Force-remove before switching profiles:
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
## 4. Host.IntegrationTests real-mode harness
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
| Dep | Container | Endpoint | Notes |
|---|---|---|---|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
```bash
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
```
## 5. Live-gate recipes (env-gated destructive tests)
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
gated by env vars and always restores the shared resource in a `finally`.
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
```bash
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
```
### 5b. LDAP GLAuth outage/recovery — **RETIRED 2026-07-31 (#520)**
`LdapGlauthOutageLiveTests` and the directory-outage circuit it drove were deleted with
`LdapOpcUaUserAuthenticator`. The OPC UA data plane authenticates with **API keys**, not LDAP, so no
LDAP bind runs inside the SDK's synchronous impersonation callback any more — which is the only
reason that circuit, its wall-clock bound, and its concurrency cap existed. LDAP remains only on the
AdminUI login path, where a slow directory stalls one request rather than serializing every OPC UA
session activation.
The harness GLAuth (§4) is still used by `OTOPCUA_HARNESS_USE_LDAP=1` and is unaffected.
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
```bash
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
export HISTGW_TEST_TAG=SysTimeSec
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
```
## 6. HistorianGateway — deploy for live testing
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
`docker/.env` / secrets, never in committed compose.
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
| Surface | Endpoint | Notes |
|---|---|---|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
```bash
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
docker compose -f docker-dev/docker-compose.yml down -v # reset
```
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
## 9. Deployment (production rig)
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
and a real directory (LDAP) — never the dev values in this doc.
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
Reference in New Issue View Git Blame Copy Permalink