fix(modbus-fixture): run pymodbus sim without asyncio debug mode; FC03 healthcheck + watchdog (#909)
Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes asyncio.run(run_main(), debug=True). Debug mode captures a traceback per scheduled callback and logs callbacks slower than 100 ms (the "took 0.146 seconds" lines in #909). Side by side on the shared docker host (closed-loop FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it. The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the debug-mode capacity, so a test run on top was answered late ("Cancel send, because not connected!") and timed out. - run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog thread that exits after 3 failed FC03 probes so restart: unless-stopped replaces the container (docker has no restart-on-unhealthy). - fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the compose healthcheck. - docker-compose.yml: shared anchor for every pymodbus service: new entrypoint, `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded json-file logs. - README: why, the numbers, and how to diagnose load.
This commit is contained in:
@@ -14,6 +14,8 @@ Docker Desktop and nothing else.
|
||||
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection` bind `:5020`, so only one of those runs at a time; `rtu_over_tcp` binds `:5021` and can run alongside `standard`) |
|
||||
| [`profiles/*.json`](profiles/) | Same seed-register definitions the native launcher uses — canonical source |
|
||||
| [`exception_injector.py`](exception_injector.py) | Pure-stdlib Modbus/TCP server that emits arbitrary exception codes per rule — used by the `exception_injection` profile |
|
||||
| [`run_simulator.py`](run_simulator.py) | Starts the pymodbus simulator with asyncio debug mode off, plus the FC03 watchdog (#909) |
|
||||
| [`fc3_probe.py`](fc3_probe.py) | FC03-read-with-deadline liveness probe: the compose healthcheck and the watchdog (#909) |
|
||||
|
||||
## Run
|
||||
|
||||
@@ -92,6 +94,44 @@ need a specific profile (the basic round-trip set) run under any of
|
||||
the three pymodbus-based profiles. The `exception_injection` profile
|
||||
is the only one that runs `exception_injector.py` instead of pymodbus.
|
||||
|
||||
## Robustness under load (#909)
|
||||
|
||||
The pymodbus simulator is a single asyncio loop, and it is shared: the docker-dev rig's
|
||||
Modbus drivers poll it too (49 drivers on each of central-1 and central-2 in November 2026,
|
||||
about 220 requests/s at idle). Four things keep it from getting swamped:
|
||||
|
||||
- **No asyncio debug mode.** pymodbus 3.13's `pymodbus.simulator` console script hard-codes
|
||||
`asyncio.run(..., debug=True)`. Debug mode captures a traceback for every scheduled callback
|
||||
and logs any callback slower than 100 ms ("Executing <Handle ...> took 0.146 seconds").
|
||||
Side by side on the shared host (closed-loop FC03 clients, 2 s deadline), debug mode served
|
||||
about 120-170 req/s to 100 clients with a p99 of 1.6-1.9 s and client timeouts; without it,
|
||||
about 2400 req/s with a p99 of about 135 ms. Every pymodbus service therefore starts through
|
||||
[`run_simulator.py`](run_simulator.py), which takes the same arguments and runs with
|
||||
`debug=False`. **Do not go back to the stock `pymodbus.simulator` entrypoint.**
|
||||
- **A real healthcheck.** [`fc3_probe.py`](fc3_probe.py) sends an FC03 read and passes only on
|
||||
a timely, well-formed reply (an exception reply counts). A TCP connect is not enough: the
|
||||
#909 simulator accepted connections and answered after the client had timed out.
|
||||
`docker ps` shows `(healthy)` / `(unhealthy)`.
|
||||
- **Self-heal.** Docker does not restart unhealthy containers. `run_simulator.py` runs the
|
||||
same probe on a watchdog thread every 10 s and exits after 3 consecutive failures, and
|
||||
`restart: unless-stopped` starts a fresh container. Tune it with `SIM_WATCHDOG_INTERVAL`
|
||||
(`0` disables it), `SIM_WATCHDOG_FAILURES` and `SIM_WATCHDOG_DEADLINE`.
|
||||
- **Bounded logs.** json-file `max-size: 10m`, `max-file: 3`.
|
||||
|
||||
With `restart: unless-stopped`, a container stopped by `docker compose down` or `docker stop`
|
||||
stays stopped. A container that was running when the docker daemon restarts comes back, so
|
||||
remove the old `:5020` profile container (see the profile-cycling gotcha above) before
|
||||
switching profiles.
|
||||
|
||||
If tests still time out, look at the load before the fixture. The `Cancel send, because not
|
||||
connected!` count in `docker logs` is the number of replies that arrived after the client had
|
||||
given up. Count the established connections and their owners:
|
||||
|
||||
```bash
|
||||
ssh dohertj2@10.100.0.35 'PID=$(docker inspect -f "{{.State.Pid}}" otopcua-pymodbus-standard); \
|
||||
sudo nsenter -t $PID -n ss -tnH state established "( sport = :5020 )" | awk "{print \$4}" | cut -d: -f1 | sort | uniq -c'
|
||||
```
|
||||
|
||||
## Endpoint
|
||||
|
||||
- Default: `localhost:5020`
|
||||
|
||||
Reference in New Issue
Block a user