Files
lmxopcua/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests/Docker/README.md
T
Joseph Doherty fc08892eb3 fix(modbus-fixture): run pymodbus sim without asyncio debug mode; FC03 healthcheck + watchdog (#909)
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.

- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
  thread that exits after 3 failed FC03 probes so restart: unless-stopped
  replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
  compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
  `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
  json-file logs.
- README: why, the numbers, and how to diagnose load.
2026-11-13 11:03:08 -05:00

187 lines
10 KiB
Markdown

# Modbus integration-test fixture — pymodbus simulator
The Modbus driver's integration tests talk to a
[`pymodbus`](https://pymodbus.readthedocs.io/) simulator running as a
pinned Docker container. One image, per-profile service in compose. Most
profiles bind `:5020`, so only one of *those* runs at a time; the
`rtu_over_tcp` profile binds `:5021` and is designed to co-run alongside
`standard`. Docker is the only supported launch path — a fresh clone needs
Docker Desktop and nothing else.
| File | Purpose |
|---|---|
| [`Dockerfile`](Dockerfile) | `python:3.12-slim-bookworm` + `pymodbus[simulator]==3.13.0` + every profile JSON + `exception_injector.py` |
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection` bind `:5020`, so only one of those runs at a time; `rtu_over_tcp` binds `:5021` and can run alongside `standard`) |
| [`profiles/*.json`](profiles/) | Same seed-register definitions the native launcher uses — canonical source |
| [`exception_injector.py`](exception_injector.py) | Pure-stdlib Modbus/TCP server that emits arbitrary exception codes per rule — used by the `exception_injection` profile |
| [`run_simulator.py`](run_simulator.py) | Starts the pymodbus simulator with asyncio debug mode off, plus the FC03 watchdog (#909) |
| [`fc3_probe.py`](fc3_probe.py) | FC03-read-with-deadline liveness probe: the compose healthcheck and the watchdog (#909) |
## Run
From the repo root:
```powershell
# Build + start the standard profile
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile standard up
# DL205 quirks
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile dl205 up
# Mitsubishi MELSEC quirks
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile mitsubishi up
# Siemens S7-1500 MB_SERVER quirks
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile s7_1500 up
# Exception-injection — end-to-end coverage of every Modbus exception code
# (01/02/03/04/05/06/0A/0B), not just the 02 + 03 pymodbus emits naturally
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile exception_injection up
```
Detached + stop:
```powershell
docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 up -d
docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 down
```
Only one of the `:5020` profiles binds at a time; switch by stopping the
current service + starting another. (`rtu_over_tcp` is the exception — it
binds `:5021` and co-runs with `standard`.) The integration tests
discriminate by a separate `MODBUS_SIM_PROFILE` env var so they skip
correctly when the wrong profile is live.
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
> *separate compose service* selected by `--profile`, so a plain
> `docker compose down` (no `--profile`) does **not** stop the
> currently-running profile container — it keeps holding `:5020`. Start
> another profile and the new container silently fails to bind the port
> (compose still reports "Started"); every test then hits the **stale**
> profile's data and fails with wrong values / Illegal-Data-Address. The
> symptom is deceptive: reads succeed but return the previous profile's
> seed (e.g. Mitsubishi's `D0` reads DL205's `0xCAFE`). Always tear the
> old container down **by name / with its profile** before bringing up the
> next, and verify the new container actually published `:5020`:
>
> ```bash
> # reliable cycle (force-remove ALL modbus containers, then bring one up)
> docker rm -f $(docker ps -aq --filter name=otopcua-modbus --filter name=otopcua-pymodbus)
> docker compose --profile <profile> up -d --force-recreate
> docker ps --filter name=otopcua --format '{{.Names}} {{.Ports}}' # MUST show 0.0.0.0:5020->5020
> ```
>
> The `exception_injection` service uses a different container name
> (`otopcua-modbus-exception-injector`, not `otopcua-pymodbus-*`), so a
> name-filtered `docker rm` that only matches `otopcua-pymodbus` will miss
> it — the `--force-recreate` above covers that case.
### Profile coverage matrix
The two general-purpose profiles cover disjoint test sets. A full pass
of the integration suite requires running both — serially on a single
docker host (the `:5020` collision), or in parallel on two hosts.
| Job | Bring up | Env to set | Expected outcome |
|---|---|---|---|
| `modbus-standard` | `lmxopcua-fix up modbus standard` | unset `MODBUS_SIM_PROFILE` (or set to `standard`) | Standard round-trip + AddressingGrammar suites pass; `ExceptionInjectionTests` (32 rows) skip with `MODBUS_SIM_PROFILE != exception_injection`. |
| `modbus-exception` | `lmxopcua-fix up modbus exception_injection` | `MODBUS_SIM_PROFILE=exception_injection` | `ExceptionInjectionTests` (32 rows) pass against the per-`(fc,address)` rule set; standard-profile suites (round-trip, AddressingGrammar) skip. |
The DL205 / Mitsubishi / S7-1500 profiles are similar — each gates its
own quirks suite via `MODBUS_SIM_PROFILE=<profile>`. Tests that don't
need a specific profile (the basic round-trip set) run under any of
the three pymodbus-based profiles. The `exception_injection` profile
is the only one that runs `exception_injector.py` instead of pymodbus.
## Robustness under load (#909)
The pymodbus simulator is a single asyncio loop, and it is shared: the docker-dev rig's
Modbus drivers poll it too (49 drivers on each of central-1 and central-2 in November 2026,
about 220 requests/s at idle). Four things keep it from getting swamped:
- **No asyncio debug mode.** pymodbus 3.13's `pymodbus.simulator` console script hard-codes
`asyncio.run(..., debug=True)`. Debug mode captures a traceback for every scheduled callback
and logs any callback slower than 100 ms ("Executing <Handle ...> took 0.146 seconds").
Side by side on the shared host (closed-loop FC03 clients, 2 s deadline), debug mode served
about 120-170 req/s to 100 clients with a p99 of 1.6-1.9 s and client timeouts; without it,
about 2400 req/s with a p99 of about 135 ms. Every pymodbus service therefore starts through
[`run_simulator.py`](run_simulator.py), which takes the same arguments and runs with
`debug=False`. **Do not go back to the stock `pymodbus.simulator` entrypoint.**
- **A real healthcheck.** [`fc3_probe.py`](fc3_probe.py) sends an FC03 read and passes only on
a timely, well-formed reply (an exception reply counts). A TCP connect is not enough: the
#909 simulator accepted connections and answered after the client had timed out.
`docker ps` shows `(healthy)` / `(unhealthy)`.
- **Self-heal.** Docker does not restart unhealthy containers. `run_simulator.py` runs the
same probe on a watchdog thread every 10 s and exits after 3 consecutive failures, and
`restart: unless-stopped` starts a fresh container. Tune it with `SIM_WATCHDOG_INTERVAL`
(`0` disables it), `SIM_WATCHDOG_FAILURES` and `SIM_WATCHDOG_DEADLINE`.
- **Bounded logs.** json-file `max-size: 10m`, `max-file: 3`.
With `restart: unless-stopped`, a container stopped by `docker compose down` or `docker stop`
stays stopped. A container that was running when the docker daemon restarts comes back, so
remove the old `:5020` profile container (see the profile-cycling gotcha above) before
switching profiles.
If tests still time out, look at the load before the fixture. The `Cancel send, because not
connected!` count in `docker logs` is the number of replies that arrived after the client had
given up. Count the established connections and their owners:
```bash
ssh dohertj2@10.100.0.35 'PID=$(docker inspect -f "{{.State.Pid}}" otopcua-pymodbus-standard); \
sudo nsenter -t $PID -n ss -tnH state established "( sport = :5020 )" | awk "{print \$4}" | cut -d: -f1 | sort | uniq -c'
```
## Endpoint
- Default: `localhost:5020`
- Override with `MODBUS_SIM_ENDPOINT` (e.g. a real PLC on `:502`).
## Run the integration tests
In a separate shell with one profile live:
```powershell
cd C:\Users\dohertj2\Desktop\lmxopcua
dotnet test tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests
```
`ModbusSimulatorFixture` probes `localhost:5020` at collection init +
records a `SkipReason` when unreachable, so tests stay green on a fresh
clone without Docker running.
## Exception injection
pymodbus's simulator naturally emits only Modbus exception codes `0x02`
(Illegal Data Address, on reads outside its configured ranges) and
`0x03` (Illegal Data Value, on over-length requests). The driver's
`MapModbusExceptionToStatus` table translates eight codes: `0x01`,
`0x02`, `0x03`, `0x04`, `0x05`, `0x06`, `0x0A`, `0x0B`. Unit tests
lock the translation function; the integration side previously only
proved the wire-to-status path for `0x02`.
The `exception_injection` profile runs
[`exception_injector.py`](exception_injector.py) — a tiny standalone
Modbus/TCP server written against the Python stdlib (zero
dependencies outside what's in the base image). It speaks the wire
protocol directly (FC 01/02/03/04/05/06/15/16) and looks up each
incoming `(fc, address)` against the rules in
[`profiles/exception_injection.json`](profiles/exception_injection.json);
a matching rule makes the server reply with
`[fc | 0x80, exception_code]` instead of the normal response.
Current rules (see the JSON file for the canonical list):
- `FC03 @1000..1007` — one per exception code (`0x01`/`0x02`/`0x03`/`0x04`/`0x05`/`0x06`/`0x0A`/`0x0B`)
- `FC06 @2000..2001` — `0x04` Server Failure, `0x06` Server Busy (write-path coverage)
- `FC16 @3000` — `0x04` Server Failure (multi-register write path)
Adding more coverage is append-only: drop a new `{fc, address,
exception, description}` entry into the JSON, restart the service,
add an `[InlineData]` row in `ExceptionInjectionTests`.
## References
- [`docs/drivers/Modbus-Test-Fixture.md`](../../../docs/drivers/Modbus-Test-Fixture.md) — coverage map + gap inventory
- [`docs/v2/dev-environment.md`](../../../docs/v2/dev-environment.md) §Docker fixtures — full fixture inventory