fc08892eb3
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes asyncio.run(run_main(), debug=True). Debug mode captures a traceback per scheduled callback and logs callbacks slower than 100 ms (the "took 0.146 seconds" lines in #909). Side by side on the shared docker host (closed-loop FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it. The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the debug-mode capacity, so a test run on top was answered late ("Cancel send, because not connected!") and timed out. - run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog thread that exits after 3 failed FC03 probes so restart: unless-stopped replaces the container (docker has no restart-on-unhealthy). - fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the compose healthcheck. - docker-compose.yml: shared anchor for every pymodbus service: new entrypoint, `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded json-file logs. - README: why, the numbers, and how to diagnose load.
187 lines
10 KiB
Markdown
187 lines
10 KiB
Markdown
# Modbus integration-test fixture — pymodbus simulator
|
|
|
|
The Modbus driver's integration tests talk to a
|
|
[`pymodbus`](https://pymodbus.readthedocs.io/) simulator running as a
|
|
pinned Docker container. One image, per-profile service in compose. Most
|
|
profiles bind `:5020`, so only one of *those* runs at a time; the
|
|
`rtu_over_tcp` profile binds `:5021` and is designed to co-run alongside
|
|
`standard`. Docker is the only supported launch path — a fresh clone needs
|
|
Docker Desktop and nothing else.
|
|
|
|
| File | Purpose |
|
|
|---|---|
|
|
| [`Dockerfile`](Dockerfile) | `python:3.12-slim-bookworm` + `pymodbus[simulator]==3.13.0` + every profile JSON + `exception_injector.py` |
|
|
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection` bind `:5020`, so only one of those runs at a time; `rtu_over_tcp` binds `:5021` and can run alongside `standard`) |
|
|
| [`profiles/*.json`](profiles/) | Same seed-register definitions the native launcher uses — canonical source |
|
|
| [`exception_injector.py`](exception_injector.py) | Pure-stdlib Modbus/TCP server that emits arbitrary exception codes per rule — used by the `exception_injection` profile |
|
|
| [`run_simulator.py`](run_simulator.py) | Starts the pymodbus simulator with asyncio debug mode off, plus the FC03 watchdog (#909) |
|
|
| [`fc3_probe.py`](fc3_probe.py) | FC03-read-with-deadline liveness probe: the compose healthcheck and the watchdog (#909) |
|
|
|
|
## Run
|
|
|
|
From the repo root:
|
|
|
|
```powershell
|
|
# Build + start the standard profile
|
|
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile standard up
|
|
|
|
# DL205 quirks
|
|
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile dl205 up
|
|
|
|
# Mitsubishi MELSEC quirks
|
|
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile mitsubishi up
|
|
|
|
# Siemens S7-1500 MB_SERVER quirks
|
|
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile s7_1500 up
|
|
|
|
# Exception-injection — end-to-end coverage of every Modbus exception code
|
|
# (01/02/03/04/05/06/0A/0B), not just the 02 + 03 pymodbus emits naturally
|
|
docker compose -f tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests\Docker\docker-compose.yml --profile exception_injection up
|
|
```
|
|
|
|
Detached + stop:
|
|
|
|
```powershell
|
|
docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 up -d
|
|
docker compose -f tests\...\Docker\docker-compose.yml --profile dl205 down
|
|
```
|
|
|
|
Only one of the `:5020` profiles binds at a time; switch by stopping the
|
|
current service + starting another. (`rtu_over_tcp` is the exception — it
|
|
binds `:5021` and co-runs with `standard`.) The integration tests
|
|
discriminate by a separate `MODBUS_SIM_PROFILE` env var so they skip
|
|
correctly when the wrong profile is live.
|
|
|
|
> **⚠️ Profile-cycling gotcha (bit us in the 2026-07 sweep — see
|
|
> `archreview/plans/INTEGRATION-SWEEP-STATUS.md`).** Each profile is a
|
|
> *separate compose service* selected by `--profile`, so a plain
|
|
> `docker compose down` (no `--profile`) does **not** stop the
|
|
> currently-running profile container — it keeps holding `:5020`. Start
|
|
> another profile and the new container silently fails to bind the port
|
|
> (compose still reports "Started"); every test then hits the **stale**
|
|
> profile's data and fails with wrong values / Illegal-Data-Address. The
|
|
> symptom is deceptive: reads succeed but return the previous profile's
|
|
> seed (e.g. Mitsubishi's `D0` reads DL205's `0xCAFE`). Always tear the
|
|
> old container down **by name / with its profile** before bringing up the
|
|
> next, and verify the new container actually published `:5020`:
|
|
>
|
|
> ```bash
|
|
> # reliable cycle (force-remove ALL modbus containers, then bring one up)
|
|
> docker rm -f $(docker ps -aq --filter name=otopcua-modbus --filter name=otopcua-pymodbus)
|
|
> docker compose --profile <profile> up -d --force-recreate
|
|
> docker ps --filter name=otopcua --format '{{.Names}} {{.Ports}}' # MUST show 0.0.0.0:5020->5020
|
|
> ```
|
|
>
|
|
> The `exception_injection` service uses a different container name
|
|
> (`otopcua-modbus-exception-injector`, not `otopcua-pymodbus-*`), so a
|
|
> name-filtered `docker rm` that only matches `otopcua-pymodbus` will miss
|
|
> it — the `--force-recreate` above covers that case.
|
|
|
|
### Profile coverage matrix
|
|
|
|
The two general-purpose profiles cover disjoint test sets. A full pass
|
|
of the integration suite requires running both — serially on a single
|
|
docker host (the `:5020` collision), or in parallel on two hosts.
|
|
|
|
| Job | Bring up | Env to set | Expected outcome |
|
|
|---|---|---|---|
|
|
| `modbus-standard` | `lmxopcua-fix up modbus standard` | unset `MODBUS_SIM_PROFILE` (or set to `standard`) | Standard round-trip + AddressingGrammar suites pass; `ExceptionInjectionTests` (32 rows) skip with `MODBUS_SIM_PROFILE != exception_injection`. |
|
|
| `modbus-exception` | `lmxopcua-fix up modbus exception_injection` | `MODBUS_SIM_PROFILE=exception_injection` | `ExceptionInjectionTests` (32 rows) pass against the per-`(fc,address)` rule set; standard-profile suites (round-trip, AddressingGrammar) skip. |
|
|
|
|
The DL205 / Mitsubishi / S7-1500 profiles are similar — each gates its
|
|
own quirks suite via `MODBUS_SIM_PROFILE=<profile>`. Tests that don't
|
|
need a specific profile (the basic round-trip set) run under any of
|
|
the three pymodbus-based profiles. The `exception_injection` profile
|
|
is the only one that runs `exception_injector.py` instead of pymodbus.
|
|
|
|
## Robustness under load (#909)
|
|
|
|
The pymodbus simulator is a single asyncio loop, and it is shared: the docker-dev rig's
|
|
Modbus drivers poll it too (49 drivers on each of central-1 and central-2 in November 2026,
|
|
about 220 requests/s at idle). Four things keep it from getting swamped:
|
|
|
|
- **No asyncio debug mode.** pymodbus 3.13's `pymodbus.simulator` console script hard-codes
|
|
`asyncio.run(..., debug=True)`. Debug mode captures a traceback for every scheduled callback
|
|
and logs any callback slower than 100 ms ("Executing <Handle ...> took 0.146 seconds").
|
|
Side by side on the shared host (closed-loop FC03 clients, 2 s deadline), debug mode served
|
|
about 120-170 req/s to 100 clients with a p99 of 1.6-1.9 s and client timeouts; without it,
|
|
about 2400 req/s with a p99 of about 135 ms. Every pymodbus service therefore starts through
|
|
[`run_simulator.py`](run_simulator.py), which takes the same arguments and runs with
|
|
`debug=False`. **Do not go back to the stock `pymodbus.simulator` entrypoint.**
|
|
- **A real healthcheck.** [`fc3_probe.py`](fc3_probe.py) sends an FC03 read and passes only on
|
|
a timely, well-formed reply (an exception reply counts). A TCP connect is not enough: the
|
|
#909 simulator accepted connections and answered after the client had timed out.
|
|
`docker ps` shows `(healthy)` / `(unhealthy)`.
|
|
- **Self-heal.** Docker does not restart unhealthy containers. `run_simulator.py` runs the
|
|
same probe on a watchdog thread every 10 s and exits after 3 consecutive failures, and
|
|
`restart: unless-stopped` starts a fresh container. Tune it with `SIM_WATCHDOG_INTERVAL`
|
|
(`0` disables it), `SIM_WATCHDOG_FAILURES` and `SIM_WATCHDOG_DEADLINE`.
|
|
- **Bounded logs.** json-file `max-size: 10m`, `max-file: 3`.
|
|
|
|
With `restart: unless-stopped`, a container stopped by `docker compose down` or `docker stop`
|
|
stays stopped. A container that was running when the docker daemon restarts comes back, so
|
|
remove the old `:5020` profile container (see the profile-cycling gotcha above) before
|
|
switching profiles.
|
|
|
|
If tests still time out, look at the load before the fixture. The `Cancel send, because not
|
|
connected!` count in `docker logs` is the number of replies that arrived after the client had
|
|
given up. Count the established connections and their owners:
|
|
|
|
```bash
|
|
ssh dohertj2@10.100.0.35 'PID=$(docker inspect -f "{{.State.Pid}}" otopcua-pymodbus-standard); \
|
|
sudo nsenter -t $PID -n ss -tnH state established "( sport = :5020 )" | awk "{print \$4}" | cut -d: -f1 | sort | uniq -c'
|
|
```
|
|
|
|
## Endpoint
|
|
|
|
- Default: `localhost:5020`
|
|
- Override with `MODBUS_SIM_ENDPOINT` (e.g. a real PLC on `:502`).
|
|
|
|
## Run the integration tests
|
|
|
|
In a separate shell with one profile live:
|
|
|
|
```powershell
|
|
cd C:\Users\dohertj2\Desktop\lmxopcua
|
|
dotnet test tests\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Modbus.IntegrationTests
|
|
```
|
|
|
|
`ModbusSimulatorFixture` probes `localhost:5020` at collection init +
|
|
records a `SkipReason` when unreachable, so tests stay green on a fresh
|
|
clone without Docker running.
|
|
|
|
## Exception injection
|
|
|
|
pymodbus's simulator naturally emits only Modbus exception codes `0x02`
|
|
(Illegal Data Address, on reads outside its configured ranges) and
|
|
`0x03` (Illegal Data Value, on over-length requests). The driver's
|
|
`MapModbusExceptionToStatus` table translates eight codes: `0x01`,
|
|
`0x02`, `0x03`, `0x04`, `0x05`, `0x06`, `0x0A`, `0x0B`. Unit tests
|
|
lock the translation function; the integration side previously only
|
|
proved the wire-to-status path for `0x02`.
|
|
|
|
The `exception_injection` profile runs
|
|
[`exception_injector.py`](exception_injector.py) — a tiny standalone
|
|
Modbus/TCP server written against the Python stdlib (zero
|
|
dependencies outside what's in the base image). It speaks the wire
|
|
protocol directly (FC 01/02/03/04/05/06/15/16) and looks up each
|
|
incoming `(fc, address)` against the rules in
|
|
[`profiles/exception_injection.json`](profiles/exception_injection.json);
|
|
a matching rule makes the server reply with
|
|
`[fc | 0x80, exception_code]` instead of the normal response.
|
|
|
|
Current rules (see the JSON file for the canonical list):
|
|
|
|
- `FC03 @1000..1007` — one per exception code (`0x01`/`0x02`/`0x03`/`0x04`/`0x05`/`0x06`/`0x0A`/`0x0B`)
|
|
- `FC06 @2000..2001` — `0x04` Server Failure, `0x06` Server Busy (write-path coverage)
|
|
- `FC16 @3000` — `0x04` Server Failure (multi-register write path)
|
|
|
|
Adding more coverage is append-only: drop a new `{fc, address,
|
|
exception, description}` entry into the JSON, restart the service,
|
|
add an `[InlineData]` row in `ExceptionInjectionTests`.
|
|
|
|
## References
|
|
|
|
- [`docs/drivers/Modbus-Test-Fixture.md`](../../../docs/drivers/Modbus-Test-Fixture.md) — coverage map + gap inventory
|
|
- [`docs/v2/dev-environment.md`](../../../docs/v2/dev-environment.md) §Docker fixtures — full fixture inventory
|