fix(modbus-fixture): run pymodbus sim without asyncio debug mode; FC03 healthcheck + watchdog (#909)

Root cause: pymodbus 3.13's `pymodbus.simulator` console script hard-codes
asyncio.run(run_main(), debug=True). Debug mode captures a traceback per
scheduled callback and logs callbacks slower than 100 ms (the "took 0.146
seconds" lines in #909). Side by side on the shared docker host (closed-loop
FC03 clients, 2 s deadline): 100 clients got ~120-170 req/s, p99 1.6-1.9 s
with client timeouts in debug mode vs ~2400 req/s, p99 ~135 ms without it.
The shared sim also carries the docker-dev rig's 49 Modbus drivers on each of
central-1/central-2 (98 sockets, ~220 req/s at idle), which is above the
debug-mode capacity, so a test run on top was answered late ("Cancel send,
because not connected!") and timed out.

- run_simulator.py: same CLI as pymodbus.simulator, debug=False, plus a watchdog
  thread that exits after 3 failed FC03 probes so restart: unless-stopped
  replaces the container (docker has no restart-on-unhealthy).
- fc3_probe.py: stdlib FC03 read with a deadline (MBAP or RTU framing); the
  compose healthcheck.
- docker-compose.yml: shared anchor for every pymodbus service: new entrypoint,
  `project: lmxopcua` label, restart: unless-stopped, healthcheck, bounded
  json-file logs.
- README: why, the numbers, and how to diagnose load.
This commit is contained in:
Joseph Doherty
2026-11-13 11:03:08 -05:00
parent e299dc090a
commit fc08892eb3
5 changed files with 278 additions and 39 deletions
@@ -14,6 +14,8 @@ Docker Desktop and nothing else.
| [`docker-compose.yml`](docker-compose.yml) | One service per profile (`standard` / `dl205` / `mitsubishi` / `s7_1500` / `exception_injection` bind `:5020`, so only one of those runs at a time; `rtu_over_tcp` binds `:5021` and can run alongside `standard`) |
| [`profiles/*.json`](profiles/) | Same seed-register definitions the native launcher uses — canonical source |
| [`exception_injector.py`](exception_injector.py) | Pure-stdlib Modbus/TCP server that emits arbitrary exception codes per rule — used by the `exception_injection` profile |
| [`run_simulator.py`](run_simulator.py) | Starts the pymodbus simulator with asyncio debug mode off, plus the FC03 watchdog (#909) |
| [`fc3_probe.py`](fc3_probe.py) | FC03-read-with-deadline liveness probe: the compose healthcheck and the watchdog (#909) |
## Run
@@ -92,6 +94,44 @@ need a specific profile (the basic round-trip set) run under any of
the three pymodbus-based profiles. The `exception_injection` profile
is the only one that runs `exception_injector.py` instead of pymodbus.
## Robustness under load (#909)
The pymodbus simulator is a single asyncio loop, and it is shared: the docker-dev rig's
Modbus drivers poll it too (49 drivers on each of central-1 and central-2 in November 2026,
about 220 requests/s at idle). Four things keep it from getting swamped:
- **No asyncio debug mode.** pymodbus 3.13's `pymodbus.simulator` console script hard-codes
`asyncio.run(..., debug=True)`. Debug mode captures a traceback for every scheduled callback
and logs any callback slower than 100 ms ("Executing <Handle ...> took 0.146 seconds").
Side by side on the shared host (closed-loop FC03 clients, 2 s deadline), debug mode served
about 120-170 req/s to 100 clients with a p99 of 1.6-1.9 s and client timeouts; without it,
about 2400 req/s with a p99 of about 135 ms. Every pymodbus service therefore starts through
[`run_simulator.py`](run_simulator.py), which takes the same arguments and runs with
`debug=False`. **Do not go back to the stock `pymodbus.simulator` entrypoint.**
- **A real healthcheck.** [`fc3_probe.py`](fc3_probe.py) sends an FC03 read and passes only on
a timely, well-formed reply (an exception reply counts). A TCP connect is not enough: the
#909 simulator accepted connections and answered after the client had timed out.
`docker ps` shows `(healthy)` / `(unhealthy)`.
- **Self-heal.** Docker does not restart unhealthy containers. `run_simulator.py` runs the
same probe on a watchdog thread every 10 s and exits after 3 consecutive failures, and
`restart: unless-stopped` starts a fresh container. Tune it with `SIM_WATCHDOG_INTERVAL`
(`0` disables it), `SIM_WATCHDOG_FAILURES` and `SIM_WATCHDOG_DEADLINE`.
- **Bounded logs.** json-file `max-size: 10m`, `max-file: 3`.
With `restart: unless-stopped`, a container stopped by `docker compose down` or `docker stop`
stays stopped. A container that was running when the docker daemon restarts comes back, so
remove the old `:5020` profile container (see the profile-cycling gotcha above) before
switching profiles.
If tests still time out, look at the load before the fixture. The `Cancel send, because not
connected!` count in `docker logs` is the number of replies that arrived after the client had
given up. Count the established connections and their owners:
```bash
ssh dohertj2@10.100.0.35 'PID=$(docker inspect -f "{{.State.Pid}}" otopcua-pymodbus-standard); \
sudo nsenter -t $PID -n ss -tnH state established "( sport = :5020 )" | awk "{print \$4}" | cut -d: -f1 | sort | uniq -c'
```
## Endpoint
- Default: `localhost:5020`