fe981b0b7f
Ran the driver CLIs against the live docker-compose fixtures to debug
what actually works. Two real bugs surfaced:
1. `e2e-config.sample.json` pointed at the wrong simulator ports:
- Modbus: 5502 → **5020** (pymodbus compose binding)
- S7: 102 → **1102** (python-snap7, non-priv port)
- AbCip: no port → now explicit **:44818**
`test-modbus.ps1` default `-ModbusHost` also shipped with 5502;
fixed to 5020.
2. Modbus loopback was off-by-2 because pymodbus `standard.json` makes
HR[100] an auto-increment register (value ticks on every poll).
Switched `test-modbus.ps1` to **HR[200]** (scratch range in the
profile) + updated sample config `bridgeNodeId` to match.
Also fixed the AbCip probe: `-t @raw_cpu_type` was rejected by the
driver's TagPath parser ("malformed TagPath"). Probe now uses the
user-supplied `-TagPath` for every family. Works against both
ab_server and real controllers.
Verified driver-side against live containers:
- Modbus 5020: probe ✓, HR[200] write+read round-trip ✓
- AB CIP 44818: probe ✓, TestDINT write+read round-trip ✓
- S7 1102: probe ✓, DB1.DBW0 write+read round-trip ✓
## Known blocker (stages 3-5)
README now flags — and the 4 child issues under umbrella #209 track —
that `src/ZB.MOM.WW.OtOpcUa.Server/Program.cs:98-104` only registers
Galaxy + FOCAS driver factories. `DriverInstanceBootstrapper` silently
skips any `DriverType` without a registered factory, so stages 3-5
(anything crossing the OPC UA server) can't work today even with a
perfect Config DB seed. Issues #210-213 scope the factory + seed SQL
work per driver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
179 lines
7.4 KiB
Markdown
179 lines
7.4 KiB
Markdown
# E2E CLI test scripts
|
|
|
|
End-to-end black-box tests that drive each protocol through its driver CLI
|
|
and verify the resulting OPC UA address-space state through
|
|
`otopcua-cli`. They answer one question per driver:
|
|
|
|
> **If I poke the real PLC through the driver, does the running OtOpcUa
|
|
> server see the change?**
|
|
|
|
This is the acceptance gate v1 was missing — the driver-level integration
|
|
tests (`tests/.../IntegrationTests/`) confirm the driver sees the PLC, and
|
|
the OPC UA `Client.CLI.Tests` confirm the client sees the server — but
|
|
nothing glued them end-to-end. These scripts close that loop.
|
|
|
|
## Five-stage test per driver
|
|
|
|
Every per-driver script runs the same five tests. The goal is to prove
|
|
**both directions** across the bridge plus subscription delivery —
|
|
forward-only coverage would miss writable-flag drops, `IWritable`
|
|
dispatch bugs, and broken data-change notification paths where a fresh
|
|
read still returns the right value.
|
|
|
|
1. **`probe`** — driver CLI opens a session + reads a sentinel. Confirms
|
|
the simulator / PLC is reachable and speaking the protocol.
|
|
2. **Driver loopback** — write a random value via the driver CLI, read
|
|
it back via the same CLI. Confirms the driver round-trips without
|
|
involving the OPC UA server. A failure here is a driver bug, not a
|
|
server-bridge bug.
|
|
3. **Forward bridge (driver → server → client)** — write a different
|
|
random value via the driver CLI, wait `--ServerPollDelaySec` (default
|
|
3s), read the OPC UA NodeId the server publishes that tag at via
|
|
`otopcua-cli read`. Confirms reads propagate from PLC to OPC UA
|
|
client.
|
|
4. **Reverse bridge (client → server → driver)** — write a fresh random
|
|
value via `otopcua-cli write` against the same NodeId, wait
|
|
`--DriverPollDelaySec` (default 3s), read the PLC-side via the
|
|
driver CLI. Confirms writes propagate the other way — catches
|
|
writable-flag drops, ACL misconfiguration, and `IWritable` dispatch
|
|
bugs the forward test can't see.
|
|
5. **Subscribe-sees-change** — start `otopcua-cli subscribe --duration N`
|
|
in the background, give it `--SettleSec` (default 2s) to attach,
|
|
write a random value via the driver CLI, wait for the subscription
|
|
window to close, and assert the captured output mentions the new
|
|
value. Confirms the server's monitored-item + data-change path
|
|
actually fires — not just that a fresh read returns the new value.
|
|
|
|
The OtOpcUa server must already be running with a config that
|
|
(a) binds a driver instance to the same PLC the script points at, and
|
|
(b) publishes the address the script writes under a NodeId the script
|
|
knows. Those NodeIds live in `e2e-config.json` (see below). The
|
|
published tag must be **writable** — stages 4 + 5 will fail against a
|
|
read-only tag.
|
|
|
|
## Status
|
|
|
|
Stages 1 + 2 (driver-side probe + loopback) are verified end-to-end
|
|
against the pymodbus / ab_server / python-snap7 fixtures. Stages 3-5
|
|
(anything crossing the OtOpcUa server) are **blocked** on server-side
|
|
driver factory wiring:
|
|
|
|
- `src/ZB.MOM.WW.OtOpcUa.Server/Program.cs` only registers Galaxy +
|
|
FOCAS factories. `DriverInstanceBootstrapper` skips any `DriverType`
|
|
without a registered factory — so Modbus / AB CIP / AB Legacy / S7 /
|
|
TwinCAT rows in the Config DB are silently no-op'd even when the seed
|
|
is perfect.
|
|
- No Config DB seed script exists for non-Galaxy drivers; Admin UI is
|
|
currently the only path to author one.
|
|
|
|
Tracking: **#209** (umbrella) → #210 (Modbus), #211 (AB CIP), #212 (S7),
|
|
#213 (AB Legacy, also hardware-gated — #222). Each child issue lists
|
|
the factory class to write + the seed SQL shape + the verification
|
|
command.
|
|
|
|
Until those ship, stages 3-5 will fail with "read failed" (nothing
|
|
published at that NodeId) and `[FAIL]` the suite even on a running
|
|
server.
|
|
|
|
## Prereqs
|
|
|
|
1. **OtOpcUa server** running on `opc.tcp://localhost:4840` (or pass
|
|
`-OpcUaUrl` to override). The server's Config DB must define a
|
|
driver instance per protocol you want to test, bound to the matching
|
|
simulator endpoint.
|
|
2. **Per-driver simulators** running. See `docs/v2/test-data-sources.md`
|
|
for the simulator matrix — pymodbus / ab_server / python-snap7 /
|
|
opc-plc cover Modbus / AB / S7 / OPC UA Client. FOCAS and TwinCAT
|
|
have no public simulator; they are gated with env-var skip flags
|
|
below.
|
|
3. **PowerShell 7+**. The runner uses null-coalescing + `Set-StrictMode`;
|
|
the Windows-PowerShell-5.1 shell will not parse `test-all.ps1`.
|
|
4. **.NET 10 SDK**. Each script either runs `dotnet run --project
|
|
src/ZB.MOM.WW.OtOpcUa.Driver.<Name>.Cli` directly, or if
|
|
`$env:OTOPCUA_CLI_BIN` points at a publish folder, runs the pre-built
|
|
`otopcua-*.exe` from there (faster for repeat loops).
|
|
|
|
## Running
|
|
|
|
### One protocol at a time
|
|
|
|
```powershell
|
|
./scripts/e2e/test-modbus.ps1 `
|
|
-ModbusHost 127.0.0.1:5502 `
|
|
-BridgeNodeId "ns=2;s=Modbus/HR100"
|
|
```
|
|
|
|
Every per-protocol script takes the driver endpoint, the address to
|
|
write, and the OPC UA NodeId the server exposes it at.
|
|
|
|
### Full matrix
|
|
|
|
```powershell
|
|
./scripts/e2e/test-all.ps1 `
|
|
-ConfigFile ./scripts/e2e/e2e-config.json
|
|
```
|
|
|
|
The runner reads the sidecar JSON, invokes each driver's script with the
|
|
parameters from that section, and prints a `FINAL MATRIX` showing
|
|
PASS / FAIL / SKIP per driver. Any driver absent from the sidecar is
|
|
SKIP-ed rather than failing hard — useful on dev boxes that only have
|
|
one simulator up.
|
|
|
|
### Sidecar format
|
|
|
|
Copy `e2e-config.sample.json` → `e2e-config.json` and fill in the
|
|
NodeIds from **your** server's Config DB. The file is `.gitignore`-d
|
|
(each dev's NodeIds are specific to their local seed). Omit a driver
|
|
section to skip it.
|
|
|
|
## Expected pass/fail matrix (default config)
|
|
|
|
| Driver | Gate | Default state on a clean dev box |
|
|
|---|---|---|
|
|
| Modbus | — | **PASS** (pymodbus fixture) |
|
|
| AB CIP | — | **PASS** (ab_server fixture) |
|
|
| AB Legacy | `AB_LEGACY_TRUST_WIRE=1` | **SKIP** (ab_server PCCC path upstream-broken — task #222) |
|
|
| S7 | — | **PASS** (python-snap7 fixture) |
|
|
| FOCAS | `FOCAS_TRUST_WIRE=1` | **SKIP** (no public simulator — task #222 lab rig) |
|
|
| TwinCAT | `TWINCAT_TRUST_WIRE=1` | **SKIP** (needs XAR or standalone Router — task #221) |
|
|
| Phase 7 | — | **PASS** if the Modbus instance seeds a `VT_DoubledHR100` virtual tag + `AlarmHigh` scripted alarm |
|
|
|
|
Set the `*_TRUST_WIRE` env vars to `1` when you've pointed the script at
|
|
real hardware or a properly-configured simulator.
|
|
|
|
## Output
|
|
|
|
Each step prints one of:
|
|
|
|
- `[PASS] ...` — step succeeded
|
|
- `[FAIL] ...` — step failed, stdout of the failing CLI is echoed below
|
|
for diagnosis
|
|
- `[SKIP] ...` — step short-circuited (env-var gate)
|
|
- `[INFO] ...` — progress note (e.g., "waiting 3s for server-side poll")
|
|
|
|
The runner ends with a coloured summary per driver:
|
|
|
|
```
|
|
==================== FINAL MATRIX ====================
|
|
modbus PASS
|
|
abcip PASS
|
|
ablegacy SKIP (no config entry)
|
|
s7 PASS
|
|
focas SKIP (no config entry)
|
|
twincat SKIP (no config entry)
|
|
phase7 PASS
|
|
All present suites passed.
|
|
```
|
|
|
|
Non-zero exit if any present suite failed. SKIPs do not fail the run.
|
|
|
|
## Why this is separate from `dotnet test`
|
|
|
|
`dotnet test` covers driver-layer + server-layer correctness in
|
|
isolation — mocks + in-process test hosts. These e2e scripts cover the
|
|
integration seam that unit tests *can't* cover by design: a live OPC UA
|
|
server process, a live simulator, and the wire between them. Run them
|
|
before a v2 release-readiness sign-off, after a driver-layer change
|
|
that could plausibly affect the NodeManager contract, and before any
|
|
"it works on my box" handoff to QA.
|