lmxopcua/infra

Infrastructure — testing & deployment

Canonical index of every piece of infrastructure OtOpcUa's tests and deployments depend on: the shared Docker host, the driver integration fixtures, the env-gated live-gate recipes, the integration-test harness, the local dev rig, and the gateway/deploy setups. This file is the single entry point; deeper history lives in the linked docs.

Credentials below are dev-only (per decision #137) — never production values. Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).

Related deep docs

• docs/v2/dev-environment.md — full historical inventory + Windows bootstrap.
• docs/v2/Galaxy.ParityRig.md — mxaccessgw / Galaxy parity-rig runbook.
• docs/drivers/*-Test-Fixture.md — per-driver fixture coverage maps.
• ../scadaproj/infra/glauth/ — the shared GLAuth source-of-truth + deploy runbook.
• reference_histgw_local_deploy, reference_ww_historian_vm (memory) — HistorianGateway deploy detail.

---

1. Shared Linux Docker host — 10.100.0.35

All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a shared Debian host, not on the dev machine. Discover everything with the project=lmxopcua label:

ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'

• From this Mac, drive it over passwordless SSH (ssh dohertj2@10.100.0.35 'docker …'). docker -H ssh://… does not work from the Windows VM (stdio bridge hangs) — the Windows VM uses the lmxopcua-fix.ps1 helper (~/bin) instead; the Mac uses SSH directly.

• Stacks live at /opt/otopcua-<driver>/ (root-owned) or ~/otopcua-<driver>/ (user-owned, where /opt needs sudo). Current layout:

  Stack dir	Purpose
  /opt/otopcua-mssql	central SQL (always-on)
  /opt/otopcua-modbus · /opt/otopcua-abcip · /opt/otopcua-s7 · /opt/otopcua-opcuaclient · /opt/otopcua-mqtt · /opt/otopcua-mtconnect · /opt/otopcua-focas	driver fixtures
  ~/otopcua-ablegacy	driver fixture (user-owned)
  ~/otopcua-harness	Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4

• The repo's tests/.../Docker/ trees are the source of truth; the host dirs are mirrored deployments. Push changes with rsync/lmxopcua-fix sync before docker compose up.

2. Central SQL Server (always-on)

Container	otopcua-mssql (mcr.microsoft.com/mssql/server:2022-latest), stack /opt/otopcua-mssql
Endpoint	10.100.0.35:14330 → 1433
Creds (dev)	sa / OtOpcUaDev_2026!
Used by	appsettings.json ConfigDb; DB-backed test suites (*.Configuration.Tests, *.Admin.Tests, *.Server.Tests)

Nuclear reset: ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'.

3. Driver integration fixtures

Each driver's integration suite TCP-probes its fixture at collection init and skips cleanly when it isn't running. Bring one up, dotnet test, tear down. Repo compose lives under tests/Drivers/…​.IntegrationTests/Docker/.

Driver	Fixture image	Endpoint	Bring up (on the host)
Modbus	otopcua-pymodbus:3.13.0	10.100.0.35:5020	docker compose --profile <standard|dl205|mitsubishi|s7_1500> up -d
AB CIP	otopcua-ab-server:libplctag-release	10.100.0.35:44818	docker compose --profile <controllogix|compactlogix|micro800|guardlogix> up -d
AB Legacy (PCCC)	otopcua-ab-server (PCCC mode)	10.100.0.35:44818	AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d (stack ~/otopcua-ablegacy)
S7	otopcua-python-snap7	10.100.0.35:1102	docker compose --profile s7_1500 up -d
OpcUaClient	mcr.microsoft.com/iotedge/opc-plc:2.14.10	opc.tcp://10.100.0.35:50000	docker compose up -d
FOCAS	otopcua-focas-fixture (vendored focas-mock, built on the host)	10.100.0.35:8193 (published)	docker compose up -d --build (stack /opt/otopcua-focas; rsync the repo's FOCAS Docker/ dir first)
MQTT	eclipse-mosquitto:2.0.22 (+ a publisher sidecar)	10.100.0.35:8883 TLS+auth · :1883 plaintext+auth	MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d (stack /opt/otopcua-mqtt)
MQTT — Sparkplug B	otopcua-sparkplug-sim (project-owned C# edge-node simulator)	same broker; group OtOpcUaSim, edge nodes EdgeA / EdgeB, EdgeA device Filler1, ~2 s cadence	docker compose --profile sparkplug up -d (same stack). Answers rebirth NCMDs; restart it to force a fresh birth
MTConnect	mtconnect/agent:2.7.0.12 + python:3.13-alpine adapter	http://10.100.0.35:5000	docker compose up -d --wait

Endpoint overrides point a suite at a real PLC instead of the sim: MODBUS_SIM_ENDPOINT · AB_SERVER_ENDPOINT · S7_SIM_ENDPOINT · OPCUA_SIM_ENDPOINT · MQTT_FIXTURE_ENDPOINT / MQTT_FIXTURE_PLAIN_ENDPOINT · MTCONNECT_AGENT_ENDPOINT · OTOPCUA_FOCAS_SIM_ENDPOINT.

otopcua-focas-sim on the host is NOT ours (#651). It is ignitionoee's mock (created by that project's provision-oee-focas.sh for ignition-site; no published port, different defaults: big-endian timers, position exponent 3, fwlib30i64 profile). OtOpcUa's fixture is otopcua-focas-fixture; the two once shared a name and overwrote each other. Never docker rm -f, rebuild or mock_patch otopcua-focas-sim or any ignition-*/oee-* container from here. The stale ~/otopcua-focas dir on the host predates the rename — do not compose up it; retire it after /opt/otopcua-focas is deployed.

MQTT needs credentials + material, unlike the other fixtures. The broker has no anonymous fallback on either listener (deliberate — an anonymous broker would let a bad auth config pass), so it will not even start until ./gen-fixture-material.sh has written secrets/ (password file + CA + server cert/key, gitignored) and MQTT_FIXTURE_USERNAME/MQTT_FIXTURE_PASSWORD are exported at docker compose up. The live suite additionally wants MQTT_FIXTURE_CA_CERT pointing at a local copy of the CA: scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt. It is also the only fixture whose services carry the project: lmxopcua label (see CLAUDE.md).

MTConnect is a TWO-service stack. The mtconnect/agent image ships no simulator, so a lone Agent answers /probe and reports every observation UNAVAILABLE forever. The adapter service (a stdlib SHDR feeder, Docker/adapter.py) is the data source; the Agent dials out to it. Its suite also probes over HTTP, not TCP — an Agent's port accepts connections before its device model is parsed. Full detail + the seeded device model: tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.MTConnect.IntegrationTests/Docker/README.md.

Fixture-cycling gotcha: profile-gated services can share a port; a plain docker compose down may leave a stale container bound. Force-remove before switching profiles: docker rm -f $(docker ps -aq --filter name=otopcua-<driver>) then up --force-recreate.

4. Host.IntegrationTests real-mode harness

tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests runs on EF-InMemory + a stub LDAP by default (zero infra). Real-mode swaps in SQL Server + a real GLAuth bind. Stack: ~/otopcua-harness (repo: tests/Server/.../Host.IntegrationTests/docker-compose.yml + glauth/config.toml).

Dep	Container	Endpoint	Notes
SQL	otopcua-harness-sql-1	10.100.0.35:14331 → 1433	port 14331 avoids the always-on 14330
GLAuth	otopcua-harness-ldap-1	10.100.0.35:3894 → 3893	GLAuth (swapped in for retired bitnami/openldap, PR #451); users below

Harness GLAuth dev users (glauth/config.toml, baseDN dc=zb,dc=local): serviceaccount/serviceaccount123 (search), alice/alice123 (full-access), bob/bob123 (read-only).

ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
export OTOPCUA_HARNESS_USE_SQL=1  OTOPCUA_HARNESS_USE_LDAP=1
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35        # harness uses port 3894
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'

5. Live-gate recipes (env-gated destructive tests)

These prove failure/recovery behavior against real infra and skip cleanly offline. Each is gated by env vars and always restores the shared resource in a finally.

5a. S7 connect-timeout blackhole (S7_1500ConnectTimeoutOutageTests)

Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a docker restart can't reproduce. docker pause gives the ACK-less-established shape — no sudo/VPN.

ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"

5b. LDAP GLAuth outage/recovery — RETIRED 2026-07-31 (#520)

LdapGlauthOutageLiveTests and the directory-outage circuit it drove were deleted with LdapOpcUaUserAuthenticator. The OPC UA data plane authenticates with API keys, not LDAP, so no LDAP bind runs inside the SDK's synchronous impersonation callback any more — which is the only reason that circuit, its wall-clock bound, and its concurrency cap existed. LDAP remains only on the AdminUI login path, where a slow directory stalls one request rather than serializing every OPC UA session activation.

The harness GLAuth (§4) is still used by OTOPCUA_HARNESS_USE_LDAP=1 and is unaffected.

5c. HistorianGateway LiveIntegration soak (Category=LiveIntegration) — needs VPN

Exercises the gateway read/write/alarm/EnsureTags paths against a real AVEVA historian. Requires the corporate VPN up and a running gateway (see §6). Full deploy detail: reference_histgw_local_deploy.

export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221   # local gw, h2c
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true              # required for h2c
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder   # gw ApiKeys:Mode=Disabled
export HISTGW_TEST_TAG=SysTimeSec
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"

6. HistorianGateway — deploy for live testing

The gateway sidecar runs locally in Docker (~/Desktop/HistorianGateway/docker/) pointed at the real AVEVA historian over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.

• Hosts (VPN up): historian gRPC CAP wonder-sql-vd03.zmr.zimmer.com:32565 (TLS); Runtime SQL :1433 (wonderapp/Wonderapp1, SQL login); Galaxy GR SQL wonder-app-vd03.zmr.zimmer.com:1433.
• Deploy: publish FDD on host, then docker compose up -d --build. gRPC h2c localhost:5221, dashboard/health localhost:5220, ApiKeys:Mode=Disabled. Corporate hostnames go in a gitignored docker/.env / secrets, never in committed compose.
• Known deploy fixes (committed to the gateway repo): FQDN via .env; libgssapi-krb5-2 gss-ntlmssp in the Dockerfile (historian v6 + SQL paths auth NAM\dohertj2 via Negotiate/GSSAPI); RuntimeDb on.
• Full recipe + gotchas: reference_histgw_local_deploy · historian VM: reference_ww_historian_vm.

7. docker-dev local rig — docker-dev/docker-compose.yml

A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA /run verification.

Surface	Endpoint	Notes
AdminUI	http://localhost:9200 (via Traefik)	round-robins central-1/2 — rebuild both to verify a UI change
OPC UA	opc.tcp://localhost:4840 (central-1) · :4841 (central-2)	data plane
Login	disabled (Security__Auth__DisableLogin=true)	auto-authenticated full-access admin — no sign-in needed
LDAP (data plane)	shared GLAuth 10.100.0.35:3893 (DevStubMode=false)	gates Client.CLI read/write role ops only

docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
# headless deploy: POST http://localhost:9200/api/deployments  (X-Api-Key)
docker compose -f docker-dev/docker-compose.yml down -v   # reset

8. Shared GLAuth (data-plane auth) — 10.100.0.35:3893

The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness GLAuth on :3894). Container zb-shared-glauth, baseDN dc=zb,dc=local, bind cn=serviceaccount,dc=zb,dc=local/serviceaccount123, test users password password. Source of truth + deploy runbook: ../scadaproj/infra/glauth/.

9. Deployment (production rig)

The deploy host runs the OtOpcUa server + mxaccessgw + the HistorianGateway sidecar as NSSM-wrapped services from C:\publish\. Refresh after landing changes with scripts/install/Refresh-Services.ps1 (stops reverse-dependency order → backs up → publishes → restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL) and a real directory (LDAP) — never the dev values in this doc.

The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the ZB.MOM.WW.HistorianGateway sidecar (see §6 and docs/Historian.md).

10. CI — Gitea Actions (.github/workflows/v2-ci.yml)

The one workflow (there is no .gitea/workflows/) runs on the Gitea act_runners gitea-runner / gitea-runner-2: a build job (restore, Release build, code-reviews/regen-readme.py --check) and a unit-tests job (the whole solution minus Category=E2E|LiveIntegration, trx → scripts/ci/assert-not-all-skipped.sh → trx upload). Read run logs read-only with tea api --login <login> /repos/dohertj2/lmxopcua/actions/runs (then …/runs/<id>/jobs, …/jobs/<id>/logs).

Things that are not obvious (#606):

• actions/upload-artifact is pinned to @v3 and continue-on-error. act_runner reports as GHES, and upload-artifact@v4+ refuses GHES outright (GHESNotSupportedError), which failed every run regardless of the tests. Do not "upgrade" it back to v4.
• AKKA_TEST_TIMEFACTOR=3 on the test step. The Runtime/ControlPlane actor-test harnesses (tests/Server/*/Harness/TestTimeFactor.cs) feed it to akka.test.timefactor, which dilates every TestKit wait (explicit durations included). The leg runs every assembly at once on the runner; a long tail of 2–8 s presence budgets each failed about one run in eight. Local runs stay at 1.
• Presence vs absence. A presence wait (ExpectMsg, AwaitAssert, a poll-until) is a ceiling and is sized from the suites' PresenceBudget (15 s); an absence window (ExpectNoMsg, settle-then- assert) is the assertion itself and keeps its short literal. Replace a fixed Task.Delay before an assertion with a poll for the condition.
• Never leave a script spinning. A timed-out script is abandoned, not stopped (TimedScriptEvaluator), so while(true){} pinned a core for the rest of the test process. Timeout tests use a bounded wall-clock spin (System.DateTime.UtcNow is sandbox-allowed).
• MIN_EXECUTED (assert-not-all-skipped.sh) is still the default 1; raise it once a green run is recorded (a full run executes ~12.7k tests across ~57 trx files).