Joseph Doherty
d5bd4226ee
Task 26 — the P2 milestone gate. Ran the live /run verification on an isolated
docker-dev rig (project otopcua-mqtt, ports 9210/4850-4851/14350, image built
from this branch) against the real Mosquitto TLS+auth broker and the C#
Sparkplug edge-node simulator on 10.100.0.35, and recorded the result.
The gate found three defects. Two are the same defect class the P1 gate found
twice — a hand-maintained AdminUI surface left behind by a driver-side feature —
and the third is a pre-existing cross-driver bug that only a Sparkplug flow
could surface.
1. CRITICAL, FIXED — MqttDriverForm still shipped its P1 Sparkplug PLACEHOLDER.
Switching Mode to SparkplugB rendered a "not available yet" notice and NO
Group ID field, so with Sparkplug ingest fully shipped there was still no
way to author a Sparkplug driver from the AdminUI at all. Sparkplug.GroupId
is the driver's entire subscription filter (spBv1.0/{GroupId}/#): blank ⇒
connected, Healthy, ingesting nothing. Now authors all five Sparkplug keys,
MERGES over the existing sub-object rather than replacing it (so a key a
newer driver adds inside it survives an older AdminUI), leaves it untouched
in Plain mode, and validates the group id as the topic segment it is.
Pinned by 8 MqttDriverFormModelTests cases, verified falsifiable — 8 RED
with the fix stubbed out.
2. PRE-EXISTING + CROSS-DRIVER, FIXED — every node label in the shared
DriverBrowseTree was an <a href="#">. In a Blazor Web App, blazor.web.js's
enhanced-navigation click interceptor resolves a bare "#" against
<base href="/">, so clicking ANY browse-tree node label navigated the whole
AdminUI to "/", tore down the circuit, and destroyed the hosting modal —
losing the browse session AND the tag selection. @onclick:preventDefault
does not help: it suppresses the browser's default action, not Blazor's own
interceptor. Labels are now <button type="button">. Dates to the component's
introduction (2026-05-28) and affects EVERY driver's picker; it surfaced
only now because choosing a rebirth scope is the first flow that requires
clicking a label rather than the ▶ toggle (always a button, always fine).
3. FIXED (mitigation) — the browse tree rendered exactly once, at open, and
never again, while OpenAsync returns as soon as the SUBSCRIBE is granted.
On Sparkplug that means it is almost always empty forever, because births
are never retained. Added a Refresh button that re-reads the root against
the SAME session (nothing reconnects, nothing observed is lost, the tag
selection is kept, only the armed rebirth scope is cleared).
Live gate result (all against the real broker + simulator, group OtOpcUaSim):
- driver authored end-to-end through the fixed MqttDriverForm; blob is
Mode:SparkplugB + Sparkplug.GroupId:OtOpcUaSim with enums as names
- browse: BROWSER OPEN chip, Sparkplug-only rebirth panel, tree renders
Group → EdgeNode → [Device] → Metric with the device folder Filler1 and
node-level metric leaves SIDE BY SIDE, and "Node Control/Rebirth" as ONE
leaf (the metric-name-is-not-a-topic-segment rule holding)
- rebirth: group scope names the group + the 32-node cap, Cancel published
NOTHING (simulator log unchanged); edge-node scope → "1 command published"
and the simulator logged "rebirth NCMD #1 received; republishing metadata";
group scope → "2 commands published", both edge nodes re-birthed; a device
and a metric both resolve UP to "edge node EdgeA"; the panel does NOT render
for a Plain MQTT device
- browse-commit wrote bindable tuples with no topic/address key —
{"groupId","edgeNodeId","metricName","deviceId"} for the device metric and
deviceId ABSENT (not blank) for the node-level one, datatypes inherited from
the birth (Int64 / Float)
- deployed (Sealed), and both nodes served live changing values through OPC UA
at the 2 s cadence — Good 0x00000000, no BadNodeIdUnknown
- death→STALE: stopping the simulator drove both nodes to 0x80000000 with an
empty value ~2 s later, and restarting it recovered them to Good on the new
birth (FillCount back to its birth-declared 1000, then counting)
- rediscovery fired exactly TWICE (once per authored scope: OtOpcUaSim/EdgeA
7 metrics, OtOpcUaSim/EdgeA/Filler1 3 metrics) and was then gated across
many subsequent births/rebirths — the anti-storm change gate working, and
EdgeB correctly filtered out as an unauthored scope
- tag editor: opens in SparkplugB (mode inference on reopen) with all four
descriptor fields populated; blank group → "A Sparkplug group ID is
required."; "Plant/1" rejected; "Node Control/Rebirth" ACCEPTED and saved
with the slash whole; dataType override persists and its key is ABSENT again
after selecting "(from birth certificate)"
Two things are NOT verified, and are recorded rather than claimed:
- Rediscovery is INERT in v3 and this is a platform gap, not an MQTT one:
nothing subscribes to IRediscoverable.OnRediscoveryNeeded and
DriverHostActor.HandleDiscoveredNodes hard-returns. A DBIRTH introducing a
metric does NOT change the OPC UA tree; redeploy. Verified from the driver's
log line only, exactly as far as it can be verified.
- A metric name containing "/" cannot be BROWSE-COMMITTED: the derived raw tag
Name is a RawPath segment and may not contain "/", so the spec-mandatory
"Node Control/Rebirth" is refused ("Row 3: Name must not contain '/'"). The
refusal is loud and all-or-nothing, never a silently mis-bound tag, and
Manual entry is the working path. Fixing it needs a name-sanitisation policy
with a collision answer, so it was recorded rather than guessed at.
Also left open + documented: an empty browse tree cannot arm a rebirth (the
scope comes from a tree click, and the session side already accepts a bare
{group}/{edgeNode} — only a UI path is missing); _canRebirth is captured at
browse-open; the tag editor injects the Plain-only payloadFormat key into a
Sparkplug blob (cosmetic — the factory routes on the tuple).
Suites: offline 1528 passed / 0 failed (581 Driver.Mqtt + 809 AdminUI + 138
Core.Abstractions); live 15/15 against the broker + simulator.
Docs: docs/drivers/Mqtt.md brought fully up to date for P2 (Sparkplug config
sub-object, both tag shapes, the ingest state machine, rediscovery, browse +
rebirth + Refresh, a Known-gaps table); Mqtt-Test-Fixture.md gains the simulator
and drops its "Sparkplug has no fixture" claims; infra/README.md §3 gains the
simulator row; CLAUDE.md gains the simulator endpoint facts; the tracking doc
marks MQTT/Sparkplug COMPLETE. docker-dev/docker-compose.mqtt.yml is the
isolated-rig overlay the gate ran on.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
205 lines
12 KiB
Markdown
205 lines
12 KiB
Markdown
# Infrastructure — testing & deployment
|
|
|
|
Canonical index of every piece of infrastructure OtOpcUa's tests and deployments
|
|
depend on: the shared Docker host, the driver integration fixtures, the env-gated
|
|
**live-gate** recipes, the integration-test harness, the local dev rig, and the
|
|
gateway/deploy setups. This file is the single entry point; deeper history lives in
|
|
the linked docs.
|
|
|
|
> **Credentials below are dev-only** (per decision #137) — never production values.
|
|
> Production uses Integrated Security / gMSA (SQL) and a real directory (LDAP).
|
|
|
|
**Related deep docs**
|
|
- [`docs/v2/dev-environment.md`](../docs/v2/dev-environment.md) — full historical inventory + Windows bootstrap.
|
|
- [`docs/v2/Galaxy.ParityRig.md`](../docs/v2/Galaxy.ParityRig.md) — mxaccessgw / Galaxy parity-rig runbook.
|
|
- `docs/drivers/*-Test-Fixture.md` — per-driver fixture coverage maps.
|
|
- [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/) — the shared GLAuth source-of-truth + deploy runbook.
|
|
- `reference_histgw_local_deploy`, `reference_ww_historian_vm` (memory) — HistorianGateway deploy detail.
|
|
|
|
---
|
|
|
|
## 1. Shared Linux Docker host — `10.100.0.35`
|
|
|
|
All Docker workloads (central SQL, driver fixtures, integration-harness deps) run on a
|
|
shared Debian host, **not** on the dev machine. Discover everything with the
|
|
`project=lmxopcua` label:
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'docker ps --filter label=project=lmxopcua --format "{{.Names}}\t{{.Status}}"'
|
|
```
|
|
|
|
- **From this Mac**, drive it over passwordless SSH (`ssh dohertj2@10.100.0.35 'docker …'`).
|
|
`docker -H ssh://…` does **not** work from the Windows VM (stdio bridge hangs) — the
|
|
Windows VM uses the `lmxopcua-fix.ps1` helper (`~/bin`) instead; the Mac uses SSH directly.
|
|
- **Stacks** live at `/opt/otopcua-<driver>/` (root-owned) or `~/otopcua-<driver>/` (user-owned,
|
|
where `/opt` needs sudo). Current layout:
|
|
|
|
| Stack dir | Purpose |
|
|
|---|---|
|
|
| `/opt/otopcua-mssql` | central SQL (always-on) |
|
|
| `/opt/otopcua-modbus` · `/opt/otopcua-abcip` · `/opt/otopcua-s7` · `/opt/otopcua-opcuaclient` · `/opt/otopcua-mqtt` | driver fixtures |
|
|
| `~/otopcua-ablegacy` · `~/otopcua-focas` | driver fixtures (user-owned) |
|
|
| `~/otopcua-harness` | Host.IntegrationTests real-mode deps (SQL + GLAuth) — see §4 |
|
|
|
|
- The repo's `tests/.../Docker/` trees are the **source of truth**; the host dirs are mirrored
|
|
deployments. Push changes with `rsync`/`lmxopcua-fix sync` before `docker compose up`.
|
|
|
|
## 2. Central SQL Server (always-on)
|
|
|
|
| | |
|
|
|---|---|
|
|
| Container | `otopcua-mssql` (`mcr.microsoft.com/mssql/server:2022-latest`), stack `/opt/otopcua-mssql` |
|
|
| Endpoint | `10.100.0.35:14330` → `1433` |
|
|
| Creds (dev) | `sa` / `OtOpcUaDev_2026!` |
|
|
| Used by | `appsettings.json` `ConfigDb`; DB-backed test suites (`*.Configuration.Tests`, `*.Admin.Tests`, `*.Server.Tests`) |
|
|
|
|
Nuclear reset: `ssh dohertj2@10.100.0.35 'cd /opt/otopcua-mssql && docker compose down -v && docker compose up -d'`.
|
|
|
|
## 3. Driver integration fixtures
|
|
|
|
Each driver's integration suite TCP-probes its fixture at collection init and **skips cleanly**
|
|
when it isn't running. Bring one up, `dotnet test`, tear down. Repo compose lives under
|
|
`tests/Drivers/….IntegrationTests/Docker/`.
|
|
|
|
| Driver | Fixture image | Endpoint | Bring up (on the host) |
|
|
|---|---|---|---|
|
|
| Modbus | `otopcua-pymodbus:3.13.0` | `10.100.0.35:5020` | `docker compose --profile <standard\|dl205\|mitsubishi\|s7_1500> up -d` |
|
|
| AB CIP | `otopcua-ab-server:libplctag-release` | `10.100.0.35:44818` | `docker compose --profile <controllogix\|compactlogix\|micro800\|guardlogix> up -d` |
|
|
| AB Legacy (PCCC) | `otopcua-ab-server` (PCCC mode) | `10.100.0.35:44818` | `AB_LEGACY_COMPOSE_PROFILE=<…> docker compose up -d` (stack `~/otopcua-ablegacy`) |
|
|
| S7 | `otopcua-python-snap7` | `10.100.0.35:1102` | `docker compose --profile s7_1500 up -d` |
|
|
| OpcUaClient | `mcr.microsoft.com/iotedge/opc-plc:2.14.10` | `opc.tcp://10.100.0.35:50000` | `docker compose up -d` |
|
|
| FOCAS | `otopcua-focas-sim` (vendored focas-mock) | `10.100.0.35:8193` | `docker compose up -d` (stack `~/otopcua-focas`) |
|
|
| MQTT | `eclipse-mosquitto:2.0.22` (+ a publisher sidecar) | `10.100.0.35:8883` TLS+auth · `:1883` plaintext+auth | `MQTT_FIXTURE_USERNAME=otopcua MQTT_FIXTURE_PASSWORD=<pw> docker compose up -d` (stack `/opt/otopcua-mqtt`) |
|
|
| MQTT — Sparkplug B | `otopcua-sparkplug-sim` (project-owned C# edge-node simulator) | same broker; group **`OtOpcUaSim`**, edge nodes **`EdgeA`** / **`EdgeB`**, `EdgeA` device **`Filler1`**, ~2 s cadence | `docker compose --profile sparkplug up -d` (same stack). Answers rebirth NCMDs; restart it to force a fresh birth |
|
|
|
|
**Endpoint overrides** point a suite at a real PLC instead of the sim:
|
|
`MODBUS_SIM_ENDPOINT` · `AB_SERVER_ENDPOINT` · `S7_SIM_ENDPOINT` · `OPCUA_SIM_ENDPOINT` ·
|
|
`MQTT_FIXTURE_ENDPOINT` / `MQTT_FIXTURE_PLAIN_ENDPOINT`.
|
|
|
|
> **MQTT needs credentials + material, unlike the other fixtures.** The broker has **no anonymous
|
|
> fallback on either listener** (deliberate — an anonymous broker would let a bad auth config pass),
|
|
> so it will not even start until `./gen-fixture-material.sh` has written `secrets/` (password file +
|
|
> CA + server cert/key, gitignored) and `MQTT_FIXTURE_USERNAME`/`MQTT_FIXTURE_PASSWORD` are exported
|
|
> at `docker compose up`. The live suite additionally wants `MQTT_FIXTURE_CA_CERT` pointing at a local
|
|
> copy of the CA: `scp dohertj2@10.100.0.35:/opt/otopcua-mqtt/secrets/ca.crt /tmp/mqtt-fixture-ca.crt`.
|
|
> It is also the only fixture whose services carry the `project: lmxopcua` label (see CLAUDE.md).
|
|
|
|
> **Fixture-cycling gotcha:** profile-gated services can share a port; a plain `docker compose down`
|
|
> may leave a stale container bound. Force-remove before switching profiles:
|
|
> `docker rm -f $(docker ps -aq --filter name=otopcua-<driver>)` then `up --force-recreate`.
|
|
|
|
## 4. Host.IntegrationTests real-mode harness
|
|
|
|
`tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests` runs on EF-InMemory + a stub LDAP by
|
|
default (zero infra). Real-mode swaps in **SQL Server** + a real **GLAuth** bind. Stack:
|
|
`~/otopcua-harness` (repo: `tests/Server/.../Host.IntegrationTests/docker-compose.yml` + `glauth/config.toml`).
|
|
|
|
| Dep | Container | Endpoint | Notes |
|
|
|---|---|---|---|
|
|
| SQL | `otopcua-harness-sql-1` | `10.100.0.35:14331` → 1433 | port 14331 avoids the always-on 14330 |
|
|
| GLAuth | `otopcua-harness-ldap-1` | `10.100.0.35:3894` → 3893 | GLAuth (swapped in for retired bitnami/openldap, PR #451); users below |
|
|
|
|
Harness GLAuth dev users (`glauth/config.toml`, baseDN `dc=zb,dc=local`):
|
|
`serviceaccount`/`serviceaccount123` (search), `alice`/`alice123` (full-access), `bob`/`bob123` (read-only).
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d'
|
|
export OTOPCUA_HARNESS_USE_SQL=1 OTOPCUA_HARNESS_USE_LDAP=1
|
|
export OTOPCUA_HARNESS_SQL_HOST=10.100.0.35,14331
|
|
export OTOPCUA_HARNESS_LDAP_HOST=10.100.0.35 # harness uses port 3894
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose down'
|
|
```
|
|
|
|
## 5. Live-gate recipes (env-gated destructive tests)
|
|
|
|
These prove failure/recovery behavior against **real** infra and skip cleanly offline. Each is
|
|
gated by env vars and always restores the shared resource in a `finally`.
|
|
|
|
### 5a. S7 connect-timeout blackhole (`S7_1500ConnectTimeoutOutageTests`)
|
|
Proves an S7 subscription survives a SYN-blackhole outage (frozen PLC), which a `docker restart`
|
|
can't reproduce. `docker pause` gives the ACK-less-established shape — no sudo/VPN.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd /opt/otopcua-s7 && docker compose --profile s7_1500 up -d'
|
|
export S7_TIMEOUT_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-python-snap7-s7_1500"'
|
|
export S7_TIMEOUT_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-python-snap7-s7_1500"'
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.S7.IntegrationTests --filter "FullyQualifiedName~ConnectTimeoutOutage"
|
|
```
|
|
|
|
### 5b. LDAP GLAuth outage/recovery (`LdapGlauthOutageLiveTests`)
|
|
Drives the directory-outage circuit through the production auth path against the harness GLAuth
|
|
(§4), paused/unpaused. Also verifies the GLAuth swap binds correctly.
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'cd ~/otopcua-harness && docker compose up -d ldap'
|
|
export GLAUTH_LIVE_HOST=10.100.0.35 # port defaults to 3894 (GLAUTH_LIVE_PORT)
|
|
export GLAUTH_OUTAGE_START_CMD='ssh dohertj2@10.100.0.35 "docker pause otopcua-harness-ldap-1"'
|
|
export GLAUTH_OUTAGE_STOP_CMD='ssh dohertj2@10.100.0.35 "docker unpause otopcua-harness-ldap-1"'
|
|
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests --filter "FullyQualifiedName~LdapGlauthOutageLive"
|
|
```
|
|
|
|
### 5c. HistorianGateway LiveIntegration soak (`Category=LiveIntegration`) — needs VPN
|
|
Exercises the gateway read/write/alarm/EnsureTags paths against a **real AVEVA historian**. Requires
|
|
the corporate **VPN up** and a running gateway (see §6). Full deploy detail: `reference_histgw_local_deploy`.
|
|
|
|
```bash
|
|
export HISTGW_GATEWAY_ENDPOINT=http://localhost:5221 # local gw, h2c
|
|
export HISTGW_GATEWAY_ALLOW_UNTRUSTED=true # required for h2c
|
|
export HISTGW_GATEWAY_APIKEY=histgw_placeholder_placeholder # gw ApiKeys:Mode=Disabled
|
|
export HISTGW_TEST_TAG=SysTimeSec
|
|
export HISTGW_WRITE_SANDBOX_TAG=HistGW.LiveTest.Sandbox
|
|
export HISTGW_BOOL_SANDBOX_TAG=HistGW.LiveTest.BoolSandbox
|
|
export HISTGW_ALARM_SOURCE=OtOpcUaLiveValidation
|
|
dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests --filter "Category=LiveIntegration"
|
|
```
|
|
|
|
## 6. HistorianGateway — deploy for live testing
|
|
|
|
The gateway sidecar runs locally in Docker (`~/Desktop/HistorianGateway/docker/`) pointed at the
|
|
**real AVEVA historian** over VPN. It's the backend for OtOpcUa's historian read/alarm/continuous paths.
|
|
|
|
- **Hosts (VPN up):** historian gRPC CAP `wonder-sql-vd03.zmr.zimmer.com:32565` (TLS); Runtime SQL `:1433`
|
|
(`wonderapp`/`Wonderapp1`, SQL login); Galaxy GR SQL `wonder-app-vd03.zmr.zimmer.com:1433`.
|
|
- **Deploy:** publish FDD on host, then `docker compose up -d --build`. gRPC h2c `localhost:5221`,
|
|
dashboard/health `localhost:5220`, `ApiKeys:Mode=Disabled`. Corporate hostnames go in a gitignored
|
|
`docker/.env` / secrets, never in committed compose.
|
|
- **Known deploy fixes** (committed to the gateway repo): FQDN via `.env`; `libgssapi-krb5-2 gss-ntlmssp`
|
|
in the Dockerfile (historian v6 + SQL paths auth `NAM\dohertj2` via Negotiate/GSSAPI); `RuntimeDb` on.
|
|
- Full recipe + gotchas: `reference_histgw_local_deploy` · historian VM: `reference_ww_historian_vm`.
|
|
|
|
## 7. docker-dev local rig — `docker-dev/docker-compose.yml`
|
|
|
|
A 2-node OtOpcUa cluster built from source for AdminUI / OPC UA `/run` verification.
|
|
|
|
| Surface | Endpoint | Notes |
|
|
|---|---|---|
|
|
| AdminUI | `http://localhost:9200` (via Traefik) | round-robins central-1/2 — **rebuild both** to verify a UI change |
|
|
| OPC UA | `opc.tcp://localhost:4840` (central-1) · `:4841` (central-2) | data plane |
|
|
| Login | disabled (`Security__Auth__DisableLogin=true`) | auto-authenticated full-access admin — no sign-in needed |
|
|
| LDAP (data plane) | shared GLAuth `10.100.0.35:3893` (`DevStubMode=false`) | gates Client.CLI read/write **role** ops only |
|
|
|
|
```bash
|
|
docker compose -f docker-dev/docker-compose.yml up -d --build central-1 central-2 traefik
|
|
# headless deploy: POST http://localhost:9200/api/deployments (X-Api-Key)
|
|
docker compose -f docker-dev/docker-compose.yml down -v # reset
|
|
```
|
|
|
|
## 8. Shared GLAuth (data-plane auth) — `10.100.0.35:3893`
|
|
|
|
The project-wide LDAP for docker-dev + live OPC UA data-plane auth (distinct from the harness
|
|
GLAuth on :3894). Container `zb-shared-glauth`, baseDN `dc=zb,dc=local`, bind
|
|
`cn=serviceaccount,dc=zb,dc=local`/`serviceaccount123`, test users password `password`. **Source of
|
|
truth + deploy runbook:** [`../scadaproj/infra/glauth/`](../../scadaproj/infra/glauth/).
|
|
|
|
## 9. Deployment (production rig)
|
|
|
|
The deploy host runs the OtOpcUa server + `mxaccessgw` + the HistorianGateway sidecar as
|
|
NSSM-wrapped services from `C:\publish\`. Refresh after landing changes with
|
|
`scripts/install/Refresh-Services.ps1` (stops reverse-dependency order → backs up → publishes →
|
|
restarts → smoke-verifies ports 4840/4841). Production creds use Integrated Security / gMSA (SQL)
|
|
and a real directory (LDAP) — never the dev values in this doc.
|
|
|
|
> The legacy Wonderware TCP/ArchestrA sidecar was retired; the historian backend is now the
|
|
> `ZB.MOM.WW.HistorianGateway` sidecar (see §6 and `docs/Historian.md`).
|